Friday, June 27, 2008

New DTP Ganymede Video at Eclipse Live

Hi all!

Just thought I'd let you know as part of the Ganymede release, Ian had most of the main projects create 15-20 minute videos to include on Eclipse Live to highlight what's new and cool in Ganymede.

Well, I'm happy to say that the DTP demo is live. :)

You can take a look at it here.

Please be gentle, as it's my first live demo video. :)

Thanks!
--Fitz

Wednesday, June 25, 2008

DTP T-shirts... Get your t-shirts here...

Hey all...

In honor of the Eclipse Ganymede release and DTP's 1.6 release, I put together a t-shirt design so you can share your DTP love with the world. It's not much, but it's colorful and after one revision yesterday (thanks Linda!) I think it includes a majority of DTP projects, components, and terms in a creative way.



Pick one up to share with your friends and coworkers. :)

Thanks to everyone involved with DTP's Ganymede release!

--Fitz

Tuesday, June 24, 2008

Eclipse Ganymede is Released Into the Wild! (Repost)

Hi all...

It should be common knowledge, but Eclipse Ganymede (also known as Eclipse 3.4) is being released today! As said by the Eclipse folks (thanks Ian for writing a great press release as always) - "The Ganymede Release is a coordinated release of 23 different Eclipse project teams and represents over 18 million lines of code." It's an enormous undertaking by a diverse group of people, projects, and companies, so kudos to everyone involved!

Along with Ganymede is the next major release of DTP (version 1.6). With this release, we add some new functionality and streamline some of what was already there...

One of the things that people have asked for over the last few releases has been a graphical SQL query editor. And it's finally there! Yay!



There is still a bunch of work left to do with it, but for a first release I think it works great. I'm not a big fan of hand-coding joins in SQL statements and this makes it VERY easy to do that. Just click and drag!

Beyond that, we did a ton of work on usability in DTP 1.6 also. We've streamlined the process for creating driver definitions and connection profiles to the point where it only takes a few clicks before you're connecting to and drilling into the Data Source Explorer to view your databases. So a big thank you goes out to Max at JBoss and the Zend folks and everyone else who provided valuable feedback during our prototyping and implementation of these changes.

It's been a heck of a year, but I think DTP is better and stronger than ever. We will get started shortly on our first maintenance release (1.6.1, due out in September) and planning for the next major Eclipse release (in June 2009).

As always, we depend on the community for support and guidance going forward. As you start to explore our Ganymede release, keep an eye to the future and let us know what you'd like to see us do by next June!

Thanks to everyone involved in the DTP 1.6 release -- from Sybase, Actuate, IBM, JBoss, Zend, Ingres, and all those who I can't remember at the moment. Everyone take a bow as the curtain drops on Ganymede, enjoy a brief rest, and then it'll be time to get going again!

--Fitz

How do you add your own custom driver template? (Repost)

Hi all...

Sorry there's been a bit of a lag between articles. We've been busy trying to get Ganymede out the door and start planning for the future (maintenance releases for 1.6 plus the next major release of DTP for next June). I need to ditch my crystal ball for a magic eight ball I think. :)

Anyway... This week we're going to chat about how to create a new custom driver template.

First of all, when would you want to do this? There are a few possibilities:

  1. You're creating support for a new database type not currently covered by DTP Enablement.
  2. You want to add support for a third-party driver (such as DataDirect or jTDS) for a currently supported database.
  3. You simply want to add an alternative driver template to complement an existing driver that adds properties or changes default values for use in your application(s).
That said, let's pick #1. We can use it as an example to provide functionality through this article series and eventually add some new database support to DTP Enablement.

For this example, let's work on SQLite support.

You can find a ton of information about SQLite on the SQLite home page (http://www.sqlite.org/). And you can grab the SQLite JDBC driver from the SQLiteJDBC page (http://www.zentus.com/sqlitejdbc/). So we'll grab the SQLite binaries for Windows (in my case) and the sqlitejdbc-v051-bin.tgz for this case. (You'll need to put the sqlitejdbc.dll in your JRE's or SDK's JRE bin directory to get this working.)

Typically the process I work through when deciding whether or not we need a custom connection profile for a given database is as follows:
  1. Can I create a new Generic JDBC driver definition that references the jar (sqlitejdbc-v051-native.jar in this case)? Yes.
  2. Can I then create a new Generic JDBC connection profile that uses my driver definition from (1) to connect to the database? Yes.
  3. Can I browse into the database to see schemas, tables, stored procedures, and the like? Unfortunately not in this case.


This means we need to go a step further and go through these stages:
  • Stage 1: Create a new Database Definition for our Database
  • Stage 2: Create a new Driver Template for our Database (and the associated UI)
  • Stage 3: Create a new Connection Profile for our Database
  • Stage 4: Create a Custom Catalog Loader for our Database
So let's start with Stage 1 and get Stage 2 started today...

For each database that is supported in DTP and presents its structure in the Data Source Explorer (DSE), we have to tell the base models what the database supports. This is represented by the Database Definition (or "DB Definition"). What data types does it handle? Does it handle aliases or triggers? What kinds of constraints?

The DB Definition file itself is simply an XMI file (an XML file that provides metadata for some other XML files). In this case, it maps back to an EMF model for the DB Definition.

I'm not going to go into the gory details here. But there's a good article on how to get started here (be sure to look at Scenario 2).

Basically we need to create an XMI file to tell DTP what basic properties this database adheres to. What data types does it support, does it have catalogs, and so on. Most of this information can be found in the database documentation.

To simplify the process a little, we have a sample Java file that can be customized to create a new XMI file. I've modified it somewhat to create the XMI file locally. And I'll post a zip with the necessary files on the DTP website so you can grab them at the end of this exercise.

Once that's created, you can create a plug-in wrapper called "org.eclipse.datatools.enablement.sqlite.dbdefinition". And in that plug-in wrapper we will use the org.eclipse.datatools.connectivity.sqm.core.databaseDefinition extension point to tell DTP about it. Basically the databaseDefinition extension point just maps the XMI file to a named vendor and a named version. That's how the underlying systems will locate it. (You'll see the terms vendor and version appear later as we define our driver template as well.)

So now we have a DB Definition and a plug-in wrapper for it. Cool. Now we can move to the first part of Stage 2: Creating a driver template.

To create a new driver template, we'll start by creating another plug-in. This plug-in will house all the non-UI bits and pieces we want for our SQLite connection profile. We'll call it "org.eclipse.datatools.enablement.sqlite".

In the manifest for our new SQLite plug-in, we will create a new org.eclipse.datatools.connectivity.driverExtension extension. This extension point is used to register driver template categories and driver templates within the DTP framework.

Remember how we were talking about vendor and version earlier? Well, now we're going to map some driver template categories to them.

First we'll create a "SQLite" category, which maps back to the vendor name we chose earlier and has org.eclipse.datatools.connectivity.db.driverCategory as a parent. All database drivers fall under this category in DTP so we can easily find them. We'll call our new category "SQLite" and give it an ID "org.eclipse.datatools.enablement.sqlite.driver.category".

Next, we'll create a "3.5.9" category, to map to the version we selected earlier (3.5.9 is the most recent version of SQLite I could find). This one will use our "SQLite" category as its parent. We'll call it "3.5.9", and give it an ID "org.eclipse.datatools.enablement.sqlite.3_5_9.category"

Lastly we'll create the driver template itself. We'll give it the name "SQLite JDBC Driver" (not very original, but easy to remember) and an ID "org.eclipse.datatools.enablement.sqlite.3_5_9.driver". We'll set it to our SQLite 3.5.9 parent category so it has some context, setting it to the category ID we made a second ago "org.eclipse.datatools.enablement.sqlite.3_5_9.category".

We know it needs a driver jar, so we'll provide a default jar name as "sqlitejdbc-v051-native.jar". If we get fancy later, we can provide some mechanisms to pre-populate the path to the local version of that jar, but for now we'll assume the user will be able to know where their jar is located and set it appropriately in the driver definition. (Yes, we'll talk about the "fancier" way to do this automatically later.)

Beyond that, we need to get a few key bits of information about the driver. Based on the documentation for SQLite, it appears that we require the following property values:
  • Driver Class: org.sqlite.JDBC
  • JDBC URL: jdbc:sqlite:test.db
Easy enough, right? Well, we also require a few other things for a standard driver definition:
  • Vendor: SQLite
  • Version: 3.5.9
  • Database name: TEST (we can extrapolate this from the sample URL)
  • User ID: (not applicable, so we just leave it blank)
With these basic bits and pieces, we have defined our driver template! Whew. Took a bit of work though, I know.

That said, we now have reusable bits we can take into the next part of this process, which is creating a connection profile that can use our new driver definition and driver template.

At this point we're just laying the ground work. You can find a zip with the plug-ins created during this exercise here.

So next time we'll look at creating a basic connection profile that can actually use these bits!

--Fitz

The DTP Driver Framework (Repost)

Hi all...

In this article, we're going to cover an introduction to the DTP Driver Framework. What are Driver Templates/Driver Definitions? What do we do with them? Where does the UI fit in? Where does the properties provider fit in? Can I make a Driver Template into a hat? Where does the Database Definition fit in?

All of these questions and more will be answered by the end of the article, I hope.

So let's start with an easy one. What is a Driver Template? A Driver Template is a named collection of properties needed to define a usable Driver Definition.

A Driver Definition is a real instance of a Driver Template, with specific paths to driver jars or particular information to help users of that driver to create a Connection Profile that uses it.

We'll cover Connection Profiles in another article, but note that Driver Templates and Driver Definitions are optional for Connection Profiles. We use them for JDBC Connection Profiles because they provide an easy way to manage drivers and driver jars without duplicating that information in each Connection Profile instance.

Basically we have ([] = optional):

[Driver Template -> Driver Definition] -> Connection Profile

Each Driver Template can fall into one parent category, but categories can be split up in many ways. For example, we have a Database category, and then break that down by Vendor (IBM, Sybase, etc.) and Version (DB2 8.1, ASE 15, etc.). These categories provide groupings for Templates so we can refer to a parent category of Templates in particular controls used for Connection Profile creation/editing - like the DriverListCombo. You can give the DriverListCombo a category ID (like the one for Derby driver templates for example) and it will grab all of the Driver Definitions that use Driver Templates that fall under that category or any of its child categories.

(Brief sidebar... In Europa and before, we used to display Driver Definitions on the Driver Definitions preference page in a tree that followed this scheme... Databases -> Vendor -> Version -> Driver Definition, but now in Ganymede we arrange this differently so it's easier to sort by vendor or version or category to get at what you're looking for.)

Think of a Driver Template as a basic set of properties needed for a driver. Driver Templates require a unique ID, a name, and a parent category. Though jar lists are required for managing JDBC drivers, they're not mandatory for all Driver Templates, so you can specify if an empty jar list is ok. You can also provide a default name for Driver Definitions that use the Template as well as a class that can programmatically provide values for properties when Driver Definitions are created.

In addition, you can provide a list of other properties for the Driver Template. It's quite customizable. For JDBC templates, we require:
  • Driver Class
  • Vendor
  • Version
  • Database Name
  • Connection URL
  • User ID
  • Password
As an example, the Derby Embedded JDBC Driver template for version 10.0 of Derby has the following basic properties:
  • id = org.eclipse.datatools.connectivity.db.derby.genericDriverTemplate
  • name = %DERBY_EMBEDDED_DRIVER_TEMPLATE_NAME (which resolves to "Derby Embedded JDBC Driver")
  • description = (but this may be used in the future in the Driver Definition UI as a tooltip)
  • parentCategory = org.eclipse.datatools.connectivity.db.derby.10_0.driverCategory, which is a child category of org.eclipse.datatools.connectivity.db.derby.driverCategory, which is a child category of org.eclipse.datatools.connectivity.db.driverCategory (so you can see the hierarchy - all Database drivers fall under "org.eclipse.datatools.connectivity.db.driverCategory", and then Derby further defines it to a "Derby" category and then a category for "10.0" beneath that)
  • jarList = derby.jar (multiple jars could be specified here, and this property can be further modified by the valuesProvider class)
  • createDefault = false (indicates whether we should create a default Driver Definition for this template when a new workbench is created for the first time)
  • emptyJarListIsOK = false (indicates that a jar list is required for this Derby 10.0 template)
  • valuesProvider = org.eclipse.datatools.connectivity.apache.internal.derby.driver.DerbyDriverValuesProvider101 (this is a class that implements the IDriverValuesProvider interface and is used in the Derby case to see if one of two JDBC driver file plug-in wrappers exists in the workbench - if it does, it will create a Driver Definition by default, changing the jarList to have a valid path to the jar and setting createDefault to true)
  • defaultDefinitionName = %DERBY_EMBEDDED_DRIVER_DEFAULT_INSTANCE_NAME (resolves to "Derby Embedded JDBC Driver 10.0" and is used if the valuesProvider finds the right plug-ins and creates a default Driver Definition)
Then when a Driver Definition is instantiated for that Driver Template, it uses the values from the Driver Template as defaults and allows the user to modify them. Think of this as a concrete instance of a template. Sort of like a Microsoft Word template used to create a Microsoft Word document. The template just suggests some defaults for the document, but the document is the actual file being modified.

So with our Derby example, we create an instance of the Derby Driver Template that takes those defaults and allows the user to modify the values to make the Driver Definition "valid". This means in effect that a) any jar files specified in the jar/zip files list are accessible and b) any required properties have values. We point to a concrete path for our derby.jar file, maybe modify the default JDBC url and user name for our basic installation, and we're on our way.

As we mentioned earlier, if the valuesProvider class finds an appropriate plug-in wrapper for the Derby driver, it will create the Driver Definition by default and the user won't have to modify it unless they need to tweak the defaults for their installation.

Once we have a Driver Definition defined for a given database type, we can then create a Connection Profile that uses the properties of the Driver Definition as defaults (jar list and driver class are shared between the Driver Definition and the Connection Profile), and we can specialize our Profile details further, perhaps adding some optional properties to the url, a unique port or database name/path, or providing a more specific user name/password combination.

Now... I mentioned in my introduction last week that you can do more with Driver Templates and Driver Definitions than just use them in database Connection Profiles.

What if you want to define some basic communication protocols for a particular Connection Profile to consume? For example, you might define an "e-mail" driver that doesn't have any jars, but defines the basic properties of an SMTP server vs. a POP3 server vs. some other e-mail server type. So you could have two or three Driver Definitions that get consumed by an e-mail Profile (nobody's written one yet, want to volunteer?) to access and display e-mail sitting on a server somewhere.

Or maybe you want to use DTP to help manage connections to application servers? There's a wide variety of those... JBoss or Tomcat or EAServer or any number of others... Why not come up with a common Connection Profile that consumes "application server" Driver Definitions to show... message queues or provide administration functionality to start/stop a server, or view the log for the server.

Perhaps you're playing with social networking sites and you want to come up with a common set of functionality across them... Would you be able to perhaps use OpenSocial with DTP to provide that functionality?

There are a ton of possibilities to explore.

Think of Driver Templates and Driver Definitions as helpers for Connection Profiles. They're not required, but they can provide a place to share common details among profiles so you don't have to duplicate it in a bunch of profiles. For example, what if you have a bunch of database Profiles that don't use the Driver Framework and you want to update your driver class/jar list for a new version of the JDBC driver? Depending on how you implemented it, your user might have to go into each and every Profile that references the old files/driver class and update them by hand. Or if you use the Driver Framework, you update it in one place and the very next time you connect, it gets picked up automatically by the Profile.

So what do you think? Can you think of places where it would be handy to use DTP's Connection Profile and Driver Frameworks to meet a need?

As always, holler if you have questions or think I'm making this stuff up as I go along. I value the input of the community. There's LOTS of room to explore to see where we can take the DTP frameworks and I'm happy to be along for the ride!

Next week we'll start talking about how to implement your own custom Driver Templates.
Until next time...
--Fitz

DTP Article Series Introduction (Reposted)

Hi all...

I am going to be writing about some of the cool things you can do with the Data Tools Project (DTP) and how you can set up your own connection profiles for a variety of purposes. My goal with this series of articles is to provide a baseline for more people to understand the DTP frameworks for connectivity so they may build on them or use them in any way they see that's useful.

DTP is more than databases. Anyone who's played with BIRT realizes that ODA, which is a part of DTP Connectivity, shows the potential for accessing a variety of data sources, from comma separated files and XML to the variety of databases we support today through the DTP Enablement project. Though SQL is cool, it's nice to have the ability to have a consistent interface to your data when you're working in Eclipse.

So I'm going to break this into a series of articles and try to post fairly regularly -- probably one a week or so -- to cover the various bits and pieces of the DTP Connectivity frameworks and how a developer could use these bits and pieces in their own tools.

We're going to cover Driver Templates and Driver Definitions first, which might seem a bit odd considering what I said just a couple of paragraphs ago about DTP being more than just databases. However, Sybase has used driver templates in a few ways beyond just for JDBC drivers. We used them to describe properties of a security specification that could then be expanded on by adopters and users for their own purposes. Templates can be anything you want them to be and we'll cover some possible uses of the driver framework in the first series of articles.

Once we've covered drivers, we'll start talking about catalog loaders. Unlike drivers, catalog loaders are definitely a database-specific thing at this point, leveraging the Database Definition and SQL Model from the DTP Model Base project to populate a rich EMF model with JDBC-related information about a particular database. This allows you to do a variety of cool things with the SQL model.

And lastly, but definitely not least, we'll then cover the connection profile framework. Here's another area, like the driver framework, where you're not limited to databases. A connection profile can connect to a JDBC database, a CSV file, an XML file, or whatever you'd like to browse. By integrating with the Platform's Common Navigator Framework, you can integrate with file systems or FTP providers, or application servers, or newsgroups if you want to make a newsgroup reader -- pretty much anything you can connect to and pull back information from, you can make a connection profile for.

So as you can see, there's a lot of functionality available in DTP. I'll do what I can to help write about the various aspects of DTP Connectivity that you may not have known about.

As always, if you have questions, post them in the comments, post them on the newsgroup, post them on the mailing list -- wherever it's convenient for you to do so. And we'll do our best to get them answered in a timely manner.

Thanks for your time! Next week we'll start talking about the Driver framework.

--Fitz

DTP Blog, Take 2...

Hi all...

You might have noticed some funky posts hogging the top spots on Planet Eclipse over the weekend. Sorry about that. The blogging software that Sybase is using isn't the greatest. Though it allowed me to schedule posts for the future, it didn't afford the same functionality to the RSS feeds, which messed everybody up. (Sorry Denis, Gunnar, and Ian).

However, after talking a bit with the folks handling the blogs, it was decided I should just create a separate blog in Wordpress or Blogger... Though I like Wordpress better (I use it for my personal blogs), it was having issues with getting a new free account created -- so here I am at Blogger!

Anyway... I feel bad about the switch, but hopefully this means you won't have to see my posts over and over and over again. I'm sure once is enough on Planet Eclipse for everybody. :)

I'm going to repost my articles so far on the Driver Framework, but promise I won't go any further than that.

Thanks for your patience!
--Fitz