G-Live White Paper
GDS Data Migration: Analysis and Solutions
1. Translating Drawings vs. Migrating the Database
Translating a few drawings from one system to another and migrating
an entire CAD/GIS database to a different system should never
be confused as similar tasks. They differ not just in the level
of effort involved, but also in the different kinds of issues
which come up. When translating a few drawings, your original
model remains intact. You translate a drawing in order to communicate
that drawing to another user. Migrating the entire database means
moving the entire model you're responsible for. It is not sufficient
that the graphics look the same; the new model must continue to
represent your real world at least as well as your old model,
and this requires that all intelligence and relationships
- not just the graphics - be migrated as well.
When translating a few drawings for use by another agency or company,
DXF is an appropriate exchange format - it is ASCII and is widely
acceptable as an import format. It delivers sufficient data (graphics,
layering, line and character style mapping) to get the necessary
line, arc, and textual information across. The process is the
digital equivalent of exchanging design drawings or blueprints.
However, just as a blueprint only communicates a limited amount
of data, such is also the case with DXF. DXF allows for exchanging
design drawings, but it does not supply the underlying data on
which the design is based (e.g., bore samples, existing surface
materials, cadastral data such as land value, use, and taxation).
If you would be reassigning an entire existing project to another
firm, exchanging blueprints would not be sufficient - you would
have to provide the entire underlying model - including specifications,
project files, and design development criteria. Converting your
digital model from one system to another is no different and no
less challenging.
A great deal of time has been spent collecting data on various
features of interest and storing that data in GDS to effectively
model your real world. Losing that data and the intelligence it
represents would be squandering a valuable, possibly irreplaceable
resource when moving to a new system. The new system should model
the real world at least as well as, if not better than GDS. Imagine
having to continue working with all your drawings, but with all
the attribute data removed. You can not just forget about it -
you originally put it in for a reason. All that data would have
to be manually re-entered, which is both extremely time consuming
and error prone.
What this implies is that your translation process must
be able to convert your GDS model of the world to at least as
good a model in your new system. Taking a real-world example,
a road is abstracted to a higher level than the underlying graphics.
The behavior and attributes of a road are implemented using the
available features in the system. In the case of GDS, a road may
be implemented using a combination of items, attribute data, and
linestyle. The representation is not the end product - the behavior
is. It is not sufficient that a translator convert a GDS
graphic block to a linestring in another system. What is needed
is the ability to convert your GDS model of a feature to a model
of that feature in your new system that behaves the same way it
did in GDS; that you can do 'what if' tests on; that
understands all the pertinent attributes and how they relate to
each other. This means that all attributes - graphical, non-graphical,
topological, etc. - must be considered. Meta data - that
is, data about the data - must likewise be considered. What projection
system is being used? What are the data types of the attribute
data? What is being communicated by the choice of linestyle? This
type of conversion is a level above that provided by DXF. When
migrating a GDS CAD/GIS database, you must convert the intelligent
features that constitute the model; with DXF, you are converting
primitive graphics, which are a simplified representation
of the model.
2. Transferring Features vs. Translating Graphic Primitives
One problem with most translators, GDS's included, is that they
rely on a neutral file format, such as DXF or IGES. If the file
format is rich enough to encompass all features of all systems,
it is probably so complex that no translator can adequately handle
every case. Such is often the case with IGES. On the other hand,
a format with a fairly rigid modeling scheme results in a translation
in which only the concepts that both systems share with the file
format, such as DXF, are converted, and the emphasis is on reformatting
the data. Essentially the translation deals only with the lowest
common denominator between the source system, the target system,
and the DXF format itself. Any aspect in the source system that
can not be directly converted to a DXF representation, and then
on to the target system, is lost.
Another shortcoming is that the emphasis is on translating a single
data construct in one system to a single type of data construct
in another system, without regard to whether that data construct
is the most appropriate for the feature being converted. It is
translating the underlying representation - not the real world
model. In the GDS DXF translator, all objects are converted to
AutoCAD blocks; all GDS items are converted to AutoCAD blocks;
attribute data, clusters and topological entities aren't converted
at all.
In a typical GDS site, for example, objects are commonly used
as a library symbol which gets copied, incorporated into linestyles,
etc., and also to logically group related graphics into a single
graphical entity. Many other systems do not offer this dual capability.
In order to use a library symbol, one type of construct may be
defined. In order to perform the logical grouping on a non-standard,
or 'non-library', entity, another type of construct may need to
be used. A typical translator does not give you the ability to
map certain objects to one representation, and other objects to
a different representation. Usually, the mapping of objects to
the target system construct was defined when the translator was
written and can not be altered.
And because only the graphics are converted, any 'implied' relationships
between objects are lost. For example, in GDS an object and its
associated dimension are related through some of the OCD facets.
When moving to another system, that relationship will be lost
because elements are related to each other in that system through
some different mechanism. Besides converting graphics from one
system to another, a translator must also be able to convert relationships
between two systems. Otherwise if you alter the dimensions of
an object, its dimension text will not be updated, and will therefore
be wrong.
Realistically with every CAD/GIS system, there are some features
that each have that the others don't. When you move from one system
to the other, the question asked should not be 'who has the closest
representation of a GDS object?' You have to ask 'who can provide
me the best model for my real world objects?' You are not converting
a GDS object and its blocks to some other primitives in the target
system - you translate a road in GDS to a road in system X. The
task of the translator, then, is to transform your real world
representation in GDS to the representation that you have considered
the best in the target system - and this must be possible on a
feature-by-feature basis, not a 'one size fits all' approach.
If here an object best transfers as a cell, but there as a graphic
group, you should be able to do that. Flexibility in setup and
execution is a must.
3. Access to Data During the Migration
Now that only deals with part of the translation issue. The other
part is that translation takes time - possibly a very long time
spanning a year or more. If your models and drawings number in
the tens of thousands, and if you make heavy use of auxiliary
drawings in your work, this becomes a critical issue. If it takes
a significant amount of time to complete the translation process,
two things will happen:
- Those who get converted first start designing in the new system,
while those who have not been converted continue working in
the old system. - or -
- All work stops until the entire conversion is done, in order
to avoid design drawings getting out of sync.
Option 1 is realistically the way a translation must go, but it
opens a number of consistency problems, such as the bridge group
designing a bridge in the new system, while the engineering group
has realigned that road to be 100 feet downstream in the old system.
Option 2 seems the most unrealistic, but is also the safest. This
ensures no inconsistencies such as illustrated above.
Over the entire migration process, projects or groups will be converted
piecemeal. But if those projects (or groups) who
have been
converted have direct, immediate access to the on-going work of
the 'unconverted' projects/groups, you greatly reduce the risk of
inconsistencies caused by using two different systems. If a group
who has not been converted, but who influences a group
who
has been converted makes a change, their updates should be automatically
available, with no intervention; without the need for someone to
keep track of when changes were made where, and then re-execute
the translation. This is analogous to the way updates are available
to others by using GDS auxiliary files. Because many groups are
inter-related, an 'unconverted' group may need to reference work
done by a converted group. In this case, using a mechanism like
DXF is appropriate since the entire model is not being translated
- just a view of the model to be used as a background.
4. Impediments to Using GDS-Supplied Translators
GDS users have a unique set of requirements. We are used to much
more flexibility than other CAD/GIS systems have offered in the
past in choosing how we want to represent things. GDS was always
well ahead of the curve - we could choose to model features as
objects, clusters, items, topological polygons, chains, or nodes.
Non-graphical attributes could be stored as part of an OCD/SCD,
as attribute data, or in an external database. GDS probably has
the richest linestyle capabilities around.
But now the rest of the world has caught up. Many vendors now
offer topology; they may have unique feature naming ala OCD's;
most have the programming interfaces like XBasic and EPI.
The problems are that GDS's out-bound translation capabilities
have not kept up, or that the available intermediate formats,
such as DXF, just don't have the support for some of the GDS constructs.
Topology can not be transferred via DXF, nor can attribute data.
Topological entities, such as polygons, could theoretically be
transferred as AutoCAD blocks, but the translator doesn't do that.
You would have to dismantle the structure so that each item in
the structure is passed as a separate block. But then the item
name is the identifier - not the original SCD - so there is no
way of reassembling those blocks back together on the other system
to re-create the original polygon. When GDS had introduced a new
feature, we as users made use of it. Unfortunately the GDS translators
did not keep pace. Conversely, the format of DXF files has changed
between various releases of AutoCAD. In particular, the format
changed with the introduction of R13, and there are still some
questions regarding R14. GDS is still continuing to output in
version R11, and there is no guarantee that this version will
be supported in the future, thus potentially breaking
any workflow in progress.
5. Requirements For Successful Migration
To overcome those limitations, a translator must meet 3 criteria:
- It should not rely on an intermediate, neutral
format, and if it does, that format must have at least the richness
of features that both the source and target systems have.
- Every piece of data and every representation must be available.
If the translation process can not get access to attribute data,
part of your model is simply lost.
- There must be a user-configurable way of mapping various features
in the source system to any of the available entities in the
target system - on an individual basis. This process is known
as semantic translation or semantic processing.
If your translation process meets all 3 criteria, than you can successfully
transfer your model. That translated model will continue to behave
as it should on the target system - with all the intelligence and
features that it had in the original system. Otherwise, you are
simply converting the underlying mathematical representation of
the graphics between the two systems, with the end result being
that all you end up with is a pretty picture - not the kind intelligence
which you can base a design on.
6. G-Live as the Solution
The G-Live product is an advanced piece of software specifically
designed to satisfy the above criteria. Files from any of a number
of vendor systems and data formats can be both read in and written
out. G-Live provides a semantic translator for specifying how
features in one system (GDS) are to be represented in the target
system. The features can be transformed into appropriate representations
based on geometry, attribute data values, or OCD/SCD facets, for
example. Any of those file formats can be opened and used as if
it were a file format native to the target system, and all of
the target system's tools are available to use with that data.
This means you can bring in data from any of a number of sources
to form a single complete model. Even the projection parameters
are brought over.
Kenneth Balk & Associates (KBA) are working with the developer
of the core translation system to build the GDS module. Rather
than using one of the GDS-supplied translators to dump the data
from GDS, an EPI program will be used to extract the data, which
is then sent over a network directly to the target system
for display. There are no intermediate files to coordinate or
get out of date. All GDS data is available to EPI, which
means that all the intelligence you assigned - attribute data,
topology, and OCD's, for example - is available to be moved to
the new system, or manipulated using the semantic processor to
be transformed into an appropriate representation on your new
system.
You can begin migrating groups, and those groups will continue
to have data from the other groups available - either as native
drawings on the target system that have already been translated,
or directly from FGB's through G-Live for those groups which have
not yet been converted. Your data now lies on the continuum of
where it is in the conversion process - not whether
it is available or not.
So the model that is formed by merging the converted
data with the unconverted GDS drawings is in fact the same
model that you had in GDS - nothing is lost, all groups
will continue working, and the entire model is available on the
new system from Day 1.
This approach really alters the traditional translation process.
Typically when translating by department, the most critical departments
are the first to be migrated, but with this approach, they must
be last. They must continue doing their designed, but their most
recent drawings must also be available to all - in both the new
and the old systems, and so they must be the last to cross
the continuum from GDS to the new system. If the conversion is
done on a project, rather than a departmental basis, the current
ongoing projects will be the first to be translated in order to
get people onto the new system as quickly as possible. However
if a project has a need to refer back to a previous archived project
which has not yet been converted, that will be possible.
7. Quality Assurance During the Migration
Finally, regardless of how good the translator is, the Quality
Assurance (QA) process is critical when migrating to a new system.
Even if great care has been spent painstakingly mapping features
in one system to features in another, work must be reviewed. Everyone
has standards to use for creating objects, what linestyles to
use, what goes on what drawing, and so on. With thousands and
thousands of drawings, somewhere, something is going to deviate
from the standard. Also, what often happens when drawings are
being created, is that some situations will arise which are not
covered by the standards. An 'ad hoc' solution is devised, but
that solution often does not make it back into the published standard.
So when using the standards to define the mapping between source
and target features, those ad hoc situations are not caught. Also
those standards may not be sufficiently detailed to have been
rigorously adhered to in the past, or for designing the
conversion mapping now.
When converting a few drawings for use by another agency, this
is not quite as critical (though still very important). If a problem
is found in the conversion, you can always go back to the original
source, fix the problem, and re-translate. But if you are migrating
all of your data, there isn't the up-to-date original source to
go back to. Groups have started using the new model in the new
system, and are maintaining that - the original data is not maintained,
so quickly becomes out-of-date. If something is found to be missing
in the translation, it either has to be re-created in the new
system, or brought back again from the old system, thereby losing
any recent changes that have been made.
With the ability to directly open a GDS drawing file in the new
system, it is tempting to save off that native file and consider
that the GDS drawing is translated. However this will lead to
serious questions as to the validity of the converted data. Without
a well thought-out transition path that is scrupulously adhered
to, confusion arises as to what has been converted, what has been
checked, and what acceptance criteria were used in the checking.
Even with the best of translators, a solid QA process is an essential
step in the migration process, and must occur as an integral part
of that process - before the data is actually
put to use and not as an optional step or an
afterthought. QA is required throughout the entire process, but
is especially important at the beginning to verify that the feature
mapping and semantic translation have the desired effect and do
in fact result in a useable model in the new system.
