Authors:
João Silva Ferreira
1
;
André Restivo
2
and
Hugo Sereno Ferreira
3
Affiliations:
1
Faculdade de Engenharia da Universidade do Porto, Portugal
;
2
Faculdade de Engenharia da Universidade do Porto, LIACC, Portugal
;
3
Faculdade de Engenharia da Universidade do Porto, INESC TEC, Portugal
Keyword(s):
Synthetic Datasets, Neural Networks, Computer Vision, Real-time, Website Generation.
Abstract:
Designers often use physical hand-drawn mockups to convey their ideas to stakeholders. Unfortunately, these
sketches do not depict the exact final look and feel of web pages, and communication errors will often occur,
resulting in prototypes that do not reflect the stakeholder’s vision. Multiple suggestions exist to tackle this
problem, mainly in the translation of visual mockups to prototypes. Some authors propose end-to-end solutions by directly generating the final code from a single (black-box) Deep Neural Network. Others propose the
use of object detectors, providing more control over the acquired elements but missing out on the mockup’s
layout. Our approach provides a real-time solution that explores: (1) how to achieve a large variety of sketches
that would look indistinguishable from something a human would draw, (2) a pipeline that clearly separates
the different responsibilities of extracting and constructing the hierarchical structure of a web mockup, (3) a
methodo
logy to segment and extract containers from mockups, (4) the usage of in-sketch annotations to provide more flexibility and control over the generated artifacts, and (5) an assessment of the synthetic dataset
impact in the ability to recognize diagrams actually drawn by humans. We start by presenting an algorithm that
is capable of generating synthetic mockups. We trained our model (N=8400, Epochs=400) and subsequently
fine-tuned it (N=74, Epochs=100) using real human-made diagrams. We accomplished a mAP of 95.37%,
with 90% of the tests taking less than 430ms on modest commodity hardware (≈ 2.3fps). We further provide
an ablation study with well-known object detectors to evaluate the synthetic dataset in isolation, showing that
the generator achieves a mAP score of 95%, ≈1.5× higher than training using hand-drawn mockups alone
(More)