CrowdProcess is a browser-...

Project Profile

CrowdProcess

project Info

CrowdProcess is a browser-powered distributed computing platform.

It helps you run a Javascript program in thousands of Web Workers around the world, in parallel, with different input data. Basically a map function.

It's in the SPMD (Single Program, Multiple Data) class of applications: "Tasks are split up and run simultaneously on multiple processors with different input in order to obtain results faster". Except that in CrowdProcess the processors are Web Workers, and they don't share any memory.

Web Workers connect to CrowdProcess from partner websites.

• • •

Basic Concepts

A job is the collection of program and data which is submitted to CrowdProcess. The program is a Javascript code with an entrypoint function named Run which will map an element of the dataset in a Web Worker to a result. The pair of each data element and program is called task and each browser connected to the platform at any given instant is inseminated by one or several tasks. The results computed from each task are returned to CrowdProcess and streamed back to the user.

  • Job: a program plus input data. It yields results.
  • Program: Javascript code with an entrypoint function named Run.
  • Task: the execution of the program with input data. A task yields a result.
  • Result: the outcome of a task.

FeaturesStupidly Scalable and Easy to Use

Each web browser is equivalent to a CPU core, so imagine having an easy to manage beast of 10k cores at your fingertips. Use it with a couple of lines.

Streaming and Realtime or Durable

Results are delivered as soon as they're computed, and you may also retrieve them at a later time.

CaveatsRunning in Web WorkersTask Running Time

Because tasks run on Web Workers, which in turn are on visitor's Web Browsers, you should plan them so that they take no more than the average visitor session lasts. The current session average lasts about 6m50s, but you'll get better results if your tasks take less time than that.

It's perfectly ok to make an extremely huge number of tasks and CrowdProcess deals with 1000 tasks that take 300 seconds as well as it deals with 10000 tasks that take 30 seconds. The second case though, will have higher chances of finishing during browser sessions.

Task Granularity

Input data will be sent to the browsers, and they have different internet connections, some slow and some fast, so as a rule of thumb, it's a good idea to plan for both and split input data of about 1mb or less.

If input data is shared across tasks, it's a very good idea to make it part of the program because less data has to travel and it's more efficient.

Security

Although every connection is SSL encrypted (from you to the browsers and back), each browser will have access to the tasks that are scheduled to it.

Tasks in browsers are ephemeral and exist only during computation time, nothing is stored ever on the browser.

If you have sensitive information, perhaps you might be interested in our Enterprise Grid solution.

Performance

Performance in CrowdProcess is affected by the number of browsers connected to the platform and their network bandwidth.

What runs well on CrowdProcess ?

Most embarrassingly parallel, CPU heavy, low I/O applications run well on CrowdProcess.

So far we have seen really good speedups on Monte-Carlo based applications. See some of our demos in portfolio variance minimization and sports statistics for inspiration.

Tags: 

Total votes: 0

About Us

Become a menber and help us show the world what Portuguese are made of

Read more...

Newsletter

Subscribe and get the latest updates, news, and more...

Latest Projects

09.10.2015
GliControl allows you to keep a daily log for diabetes. View a list of glucose records/carbohydrates/insulin. Know daily where...
05.04.2015
The Moscatel de Setúbal belongs to the quartet of the most acclaimed fortified wines of Portugal, along with Port, Madeira and...
15.11.2014
If you always dreamed of creating a small garden on the balcony and not know where to start, we suggest you take a taste with...

Auto Translate

English Dutch French German Japanese Portuguese

Languages

Contact Us

from Portugal to the world

Email: [email protected]