AiroCorp- Project Crawler

What is project crawler?

Project Crawler is an open source project by AiroCorp. This Project is intended to make the entire Internet as a working source of data using Artificial intelligence and Natural language processing. The Crawler is based on the basic functionality of a web spider, which is used to index the web links for a search engine. But instead of links, the crawler will be scrapping for content and the NLP based analysis of the content will be fetched from the sites.

This will allow the crawler to find keywords, facts and named entities from a website. Using these entities and keywords, we can create a recursive process with a predefined depth to find a lot of information about a given topic without having any static database or file. Entire Internet will work as a source of data. This will open a new way and reduce the complexity of data management and handling of the database. It will also provide new methods of using the existing information in multiple domains.

Current State

It is at very initial state. We will release the first code snippets soon. It will be a public version so anyone can download the source-code form our website or our public Git-Hub repository. However, to submit an update to the code, the user has to be registered with AiroCorp. We will review the submissions to make the new version of it. Downloading the source and binary code will always be free.

It is at very initial state. We will release the first code snippets soon. It will be a public version so anyone can download the source-code form our website or our public Git-Hub repository. However, to submit an update to the code, the user has to be registered with AiroCorp. We will review the submissions to make the new version of it. Downloading the source and binary code will always be free.

Technical Specification

The registration form, technical details and documentation will soon be available here. We are still working on the first base code for the crawler as well as the class definitions and other libraries required. If you have any suggestions or want to be the part of the team, which is working on the base (early) version with the concept itself and the methods to implement it, write us at contact@airocorp.com.