Tika File Digester

-



I got a requirement on create a module to read, persist and index the any kind of document or file content. I have been trying out several APIs and spend few hours with them and play around with them. Then I identified Apache Tika  is the best suitable API for file digesting and content extraction. It supports lots of MIME types.

Current version I have been using is Apache Tika 1.4. It supports the following document formats.


Maven dependency

<dependency>
    <groupId>org.apache.tika</groupId>
    <artifactId>tika-core</artifactId>
    <version>...</version>
  </dependency>

 <dependency>
    <groupId>org.apache.tika</groupId>
    <artifactId>tika-parsers</artifactId>
    <version>...</version>
  </dependency>


Tika as a command line utility

Tika has its own command line utility to try out the Tika. Which was the initial attraction point of my self towards Tika. 
Following command will open up a GUI to play with Tika.

usage: java -jar tika-app.jar -g



Read Files using Tika

Following java code will describes you to how to read or digest a document using Tika.

 public void fileContentRead() throws IOException, SAXException, TikaException
    {
        String fileName = "my_test.docx";
        Tika tika = new Tika();
        InputStream stream = new FileInputStream(fileName);
        System.out.println("Stream Data :" + stream.available());
        ByteArrayOutputStream os = new ByteArrayOutputStream();
        Metadata metadata = new Metadata();                                                                                                                             // Auto detect the parser according to the MIME type of the file.
        Parser parser = new AutoDetectParser();
        long start = System.currentTimeMillis();
        BodyContentHandler handler = new BodyContentHandler(os);
        InputStream content = stream;
        parser.parse(content, handler, metadata, new ParseContext());
        for (String name : metadata.names())
        {
            System.out.println(name + ":\t" + metadata.get(name));
        }
        System.out.println("\n\nContent :\n" + new String(os.toByteArray()));
        System.out.println(String.format("------------ Processing took %s                    millis\n\n",
             System.currentTimeMillis() - start));
        stream.close();
    }

The output of this method can be like:

Easy Shooting Mode : Manual
Image Type : Canon EOS 350D DIGITAL
Model : Canon EOS 350D DIGITAL
Metering Mode : Evaluative
Quality : Fine
Shutter/Auto Exposure-lock Buttons : AF/AE lock
ISO Speed Ratings : 400

Looking a head I'm focusing on integrating Apache Tika with Solr server to digest the files and automatically index the documents inside the Solr. Which is my ultimate goal.


References





Location: Singapore

Comments

Post a Comment

Popular posts from this blog

PostgreSQL bytea and oid

-
I came across with following issue when i'm trying to save the uploaded file in to the database as a byte array using hibernate, JPA 2.0  annotation with Postgre SQL. even though in this stage i'm pretty much new to Postgre SQL, It has two type of data types to represent the binary data or BLOB, since i'm using hibernate jpa annotations use the annotation @Lob to indicate the byte[] of data to persist as a binary data. @Lob   @Basic(fetch = FetchType.LAZY)  public byte[] getPicture() { return picture;  }    exception : java.sql.SQLException: ERROR: column "picture" is of type bytea but expression is of type oid   Then i understood the issue with the data types.but when the hibernate generating the table it creates the  column with data type "oid" , after removing this annotation it creates a table column with "bytea" data type. and i'm in doubt which would be the most suitable data type to use.   Then i found the difference be
Read more

Adding MySQL datasource to JBOSS AS 7

-
Retrieve the MySQL Connector The connector can be downloaded from:  http://dev.mysql.com/downloads/connector/j/ . The current version at the time of writing is 5.1.24, further sections of this guide assume that version number. Unzip the connector to your downloads folder and save the location of the main JAR in a variable which we can refer to later: Add a Module to AS 7 AS 7 uses a module system to provide isolation in class loading. We need to create a new module which contains the MySQL Connector J JAR. Move to the the AS installation directory and create the folder structure for the new module: export JBOSS_HOME =~ / Development / jboss-as-web-7.0.0.Final cd $JBOSS_HOME mkdir -p modules / com / mysql / main Copy the driver jar to the new directory and move to that directory: cp MYSQL_JAR $JBOSS_HOME / modules / com / mysql / main cd $JBOSS_HOME / modules / com / mysql / main Define the module in XML. This is the key part of the process.  v
Read more

Microservices Architecture with Spring Boot in 15mins

-
Image
1. Introduction Microservices is not a new term. It coined in 2005 by Dr Peter Rodgers then called micro web services based on SOAP. It became more popular since 2010. Micoservices allows us to break our large system into number of independent collaborating processes. Lets see below microservices architecture. 1.1 What is Microservices Architecture? Microservices architecture allows to avoid monolith application for large system. It provide loose coupling between collaborating processes which running independently in different environments with tight cohesion. So lets discuss it by an example as below. For example imagine an online shop with separate microservices for user-accounts, product-catalog order-processing and shopping carts. So these components are inevitably important for such a large online shopping portal. For online shopping system we could use following architectures. 1.2 Shopping system without Microservices (Monolith architecture) In this architecture we are
Read more