2.1 Simplifying the Operational Aspects
The workflow depicted in Figure 2 explains how the
PS delivered by the EGMS are converted into data
stored in the server database. This workflow is fully
operational and produces correct results. However, it
was conceived to process a single data set at once.
A data set, in this context, is the minimum item
that may be downloaded from the EGMS web site.
It defines, consequently, the granularity of the opera-
tions; in other words, since the workflow in Figure 2
is conceived to process one of these data sets, to fin-
ish the project, every single available data set must be
processed.
Considering that there are about 15,000 data sets
and all of them need to be processed to complete the
project, it should be clear that a lot of human inter-
vention is required. Such a situation implies that the
total time needed to process all the data will be signif-
icantly increased compared to a more automated pro-
cedure. Furthermore, the greater the human interven-
tion involved in the process, the greater the probabil-
ity of errors. Therefore, it seemed clear to the devel-
opment team that the already implemented workflow
needed to be improved to simplify the operational as-
pects of the process.
Automation, however, is conditioned by a tough
prerequisite: uniformity. The ADAfinder step, that is,
identifying ADAs using the PS as input, is a task that
requires adjusting a series of parameters that usually
depend on the input data (as, for instance, the size of
the pixel used in the input imagery, see (Barra et al.,
2017)). On the other hand, the ADAs published on
the web, should be obtained using a set of identical
parameters, since, otherwise, the results for differ-
ent geographical areas could not be compared. This
makes it possible to go for a solution where the level
of automation is much higher. The handicap is to find
a reasonable set of suitable set of parameters to run
ADAfinder for all Europe. This is a task currently un-
dergoing.
At any rate, and since automation is not only de-
sirable but possible, a new set of tools have already
been implemented to make it possible. These are:
• A script that makes possible to run ADAfinder for
all the data sets (EGMS data) stored in a single
directory. The same set of parameters will be used
for every data set. All results are stored in the
same output folder.
• A new application, automateADA2PGIS, will run
ADA2PGIS—the step that transforms the output
of ADAfinder into something suitable to be in-
serted in the server’s database—for all the input
files (ADAfinder-compatible ones) found in the
same directory. Again, all its outputs will be
stored in an unique folder.
Each time that ADA2PGIS is run, a set of out-
put files is created; some of them contain SQL (Stan-
dard Query Language) commands and the last one is
a batch (shell) file that will take care of actually per-
forming the process of inserting the ADAs included
in these SQL files into the database. Since auto-
mateADA2PGIS will run ADA2PGIS for all the in-
puts found in some directory, there will be an equiv-
alent number of outputs. This means that if the num-
ber of input data sets is n, also n batch (shell) files
will be created, each of them taking care of inserting
their respective ADAs. This number n is, in fact, the
total number of items that may be downloaded from
the EGMS, which amounts to about 15,000. Conse-
quently, the operator should run, by hand, all these
shell files to complete the process, which is, again, a
lengthy and tedious procedure, prone to human errors.
To avoid this situation, automateADA2PGIS also cre-
ate a so-called ”master batch file” that takes care of
running each of the individual batch files for every
data set. This completes the automation of the pro-
cess.
To summarize, the automation of the data produc-
tion pipeline could be (but it is not, see section 2.3)
the following one:
1. Download the whole EGMS data sets. Store these
in the same directory, namely the EGMS folder.
2. Run the ADAfinder automated script, taking as in-
put all the files stored in the EGMS folder. All
outputs are stored in the same output directory,
namely the ADAfinder folder. This step identifies
the ADAs.
3. Run the automateADA2PGIS tool, taking as in-
put the files stored in the ADAfinder folder. All
outputs are stored in the same output directory,
namely the automateADA2PGIS folder. This
step transforms the whole set of ADAs into a
format that may be used to insert them into a
database (files with SQL commands). A mas-
ter batch file is generated, controlling the execu-
tion of the individual batch files output by each
run of ADA2PGIS—which is executed by auto-
mateADA2PGIS.
4. Run the master batch file. This step will insert the
whole set of ADAs into the server’s database.
Note that the workflow above first downloads all
the data sets from the EGMS (step 1) to process them
later at once (steps 2 to 4) thanks to the high level of
automation achieved with the script to run ADAfinder
and the automateADA2PGIS tool. Of course, it would
New Challenges in the Implementation and Exploitation of a Low-Cost Web Map of the Active Deformation Areas Across Europe
195