Skip to content

Commit f7cb4f7

Browse files
TheItCrOwTheItCrOw
committed
2 parents 5aa102f + b05eadf commit f7cb4f7

1 file changed

Lines changed: 123 additions & 4 deletions

File tree

‎documentation/docs/getting-started/uima-duui.md‎

Lines changed: 123 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,16 +15,135 @@ graph LR
1515

1616
## UIMA
1717

18-
To be added.
18+
UIMA (Unstructured Information Management Architecture), is a framework designed to process and analyze large volumes of unstructured data, such as text, audio, images, and video. It provides a standardized and flexible platform to integrate various analysis components, making it easier to build complex systems for tasks like natural language processing (NLP), information extraction, and machine learning.
1919

2020
## DUUI
2121

22-
To be added.
22+
DUUI (Docker Unified UIMA Interface) is a platform designed to efficiently process large media corpora (mainly text, but offers also support for other media types such as video, images, and audio). It builds on the UIMA framework, using it to manage annotations, and leverages container technologies to integrate a variety of NLP tools. DUUI supports both horizontal scaling (distributing processing across multiple machines) and vertical scaling (optimizing resource use on a single machine), making it highly scalable for handling big data. It accommodates diverse NLP tools and programming languages, abstracting their differences through a unified interface, and ensures reproducibility by tracking processing pipelines. Additionally, DUUI provides robust monitoring and error-reporting features to manage large-scale tasks and is designed for ease of use, making advanced NLP accessible to users with varying technical expertise, including non-experts in research fields like digital humanities, biodiversity etc.
2323

2424
## Tut 1
2525

26-
To be added.
26+
This example demonstrates a simple annotation process: (1) creating a UIMA document, (2) setting its text, (3) splitting the text at the whitespace level, and then (4) adding token annotations to the UIMA document.
2727

28+
```java
29+
import org.apache.uima.UIMAException;
30+
import org.apache.uima.jcas.JCas;
31+
import org.apache.uima.jcas.JCasFactory;
32+
import org.apache.uima.jcas.tcas.Annotation;
33+
import org.apache.uima.cas.FSIterator;
34+
35+
public class JCasExample {
36+
public static void main(String[] args) {
37+
try {
38+
// Step 1: Create the JCas object
39+
JCas jc = JCasFactory.createJCas();
40+
41+
// Step 2: Set the sofa string (document text)
42+
String text = "This is a sample text: I like DUUI and UCE";
43+
jc.setDocumentText(text);
44+
45+
// Step 3: Add token annotations by splitting on whitespace
46+
String[] tokens = text.split("\\s+"); // Split by one or more whitespace characters
47+
int currentPosition = 0;
48+
49+
for (String token : tokens) {
50+
// Find the starting position of the current token in the original text
51+
int begin = text.indexOf(token, currentPosition);
52+
int end = begin + token.length();
53+
54+
// Step 4: Create an annotation for the token and add the annotation to the JCas
55+
Annotation annotation = new Annotation(jc, begin, end);
56+
annotation.addToIndexes();
57+
58+
// Update the current position to search for the next token
59+
currentPosition = end;
60+
}
61+
62+
// Step 5 (optional): Print the annotations
63+
FSIterator<Annotation> iterator = jc.getAnnotationIndex(Annotation.type).iterator();
64+
while (iterator.hasNext()) {
65+
Annotation ann = iterator.next();
66+
System.out.println("Token: " + ann.getCoveredText() + " [" + ann.getBegin() + ", " + ann.getEnd() + "]");
67+
}
68+
} catch (UIMAException e) {
69+
e.printStackTrace();
70+
}
71+
}
72+
}
73+
```
2874
## Tut 2
75+
This example demonstrates the initialization and execution of a DUUI pipeline for processing a corpus. It includes a simple language detection component to annotate the language of the documents. The example covers the following steps: (1) Define the input and output corpus paths, (2) Declare a pipeline reader, (3) Initialize a Lua context, (4) Initialize the composer, (5) Initialize drivers and add them to the pipeline, (6) Initialize components and add them to the pipeline, (7) Add a writer to the pipeline, (8) Run the pipeline.
76+
77+
```java
78+
import org.dkpro.core.api.resources.CompressionMethod;
79+
import org.dkpro.core.io.xmi.XmiWriter;
80+
import org.junit.jupiter.api.Test;
81+
import org.texttechnologylab.DockerUnifiedUIMAInterface.DUUIComposer;
82+
import org.texttechnologylab.DockerUnifiedUIMAInterface.driver.*;
83+
import org.texttechnologylab.DockerUnifiedUIMAInterface.io.DUUIAsynchronousProcessor;
84+
import org.texttechnologylab.DockerUnifiedUIMAInterface.io.DUUICollectionReader;
85+
import org.texttechnologylab.DockerUnifiedUIMAInterface.io.reader.DUUIFileReaderLazy;
86+
import org.texttechnologylab.DockerUnifiedUIMAInterface.lua.DUUILuaContext;
87+
88+
import java.io.File;
89+
90+
import static org.apache.uima.fit.factory.AnalysisEngineFactory.createEngineDescription;
91+
92+
public class PipelineTest {
93+
94+
@Test
95+
public void EUBooks() throws Exception {
2996

30-
To be added.
97+
int iWorker = 1;
98+
// Step 1: Define input path to a corpus and output path, where processed corpus should be saved
99+
String sInputPath = "/tmp/EUBook/input";
100+
String sOutputPath = "/tmp/EUBook/output";
101+
102+
String sSuffix = "xmi.bz2";
103+
104+
// Step 2: Define Reader
105+
DUUICollectionReader pReader = new DUUIFileReaderLazy(sInputPath, sSuffix, sOutputPath, ".xmi.bz2", 1);
106+
107+
// Asynchronous Reader for the Input Files
108+
DUUIAsynchronousProcessor pProcessor = new DUUIAsynchronousProcessor(pReader);
109+
new File(sOutputPath).mkdir();
110+
111+
// Step 3: Initialize Lua Context
112+
DUUILuaContext ctx = new DUUILuaContext().withJsonLibrary();
113+
114+
// Step 4: Initialize Composer
115+
DUUIComposer composer = new DUUIComposer()
116+
.withSkipVerification(true) // skik component verification
117+
.withLuaContext(ctx) // set lua context
118+
.withWorkers(iWorker); // set threads for the composer
119+
120+
// Step 5: Initialize all needed Drivers (based on what kind of components one would like to use)
121+
DUUIDockerDriver docker_driver = new DUUIDockerDriver();
122+
123+
// add drivers to the composer
124+
composer.addDriver(docker_driver);
125+
126+
// Step 6: Initialize components and add them to the composer scope
127+
DUUIPipelineComponent componentLang = new DUUIDockerDriver
128+
//DUUIPipelineComponent componentLang = new DUUIDockerDriver
129+
.Component("docker.texttechnologylab.org/languagedetection:0.5")
130+
.withImageFetching()
131+
.withScale(iWorker)
132+
.build();
133+
composer.add(componentLang);
134+
135+
// Step 7: Add a writer to the pipeline
136+
composer.add(new DUUIUIMADriver.Component(createEngineDescription(XmiWriter.class,
137+
XmiWriter.PARAM_TARGET_LOCATION, sOutputPath,
138+
XmiWriter.PARAM_PRETTY_PRINT, true,
139+
XmiWriter.PARAM_OVERWRITE, true,
140+
XmiWriter.PARAM_VERSION, "1.1",
141+
XmiWriter.PARAM_COMPRESSION, CompressionMethod.BZIP2
142+
)).withScale(iWorker).build());
143+
144+
// Step 8: Run the pipeline
145+
composer.run(pProcessor, "eubook");
146+
}
147+
}
148+
149+
```

0 commit comments

Comments
 (0)