-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathREADME
143 lines (99 loc) · 5.08 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
###############################################################################
# * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * #
###############################################################################
This document describes MCE version 1.699.
Many-Core Engine (MCE) for Perl helps enable a new level of performance by
maximizing all available cores. MCE spawns a pool of workers and therefore
does not fork a new process per each element of data. Instead, MCE follows
a bank queuing model. Imagine the line being the data and bank-tellers the
parallel workers. MCE enhances that model by adding the ability to chunk
the next n elements from the input stream to the next available worker.
MCE uses domain sockets for IPC versus pipes. One day, a physical box will
have hundreds of cores. MCE can spawn 960 workers (via fork) for ulimit re-
porting 1024 allowed for max user processes. Other parallel implementations
using pipes cannot do this (only about half of ulimit -u).
Chunking in MCE.
Input data is optional in MCE. Thus, input data is not required to run MCE.
MCE iterates over input data in chunks. Workers can be spawned prior to
creating or obtaining data. MCE is a chunking engine. It does not divide
the input equally by the number of workers. Instead, it chunks from
start till end.
Imagine input data as a highway. Now think of MCE (automobile) on the
highway. Basically, data can be larger than physical memory allows.
The chunking nature of MCE makes this possible.
input_data => '/path/to/file', ## process a file in parallel
input_data => \@array, ## process an array in parallel
input_data => \*FileHandle, ## process a filehandle in parallel
input_data => \$scalar, ## treated like a memory-based file
input_data => \&iterator, ## allows for custom input iterator
MCE can iterate over a sequence of numbers mathematically if all you need
is a numeric iterator. Check out forseq.pl and seq_demo.pl.
sequence => { begin => 1, end => 100, step => 2 },
chunk_size => 20, ## Worker receives the next 20 sequences
Use Cases for MCE.
1. Perl is not compiled with threads support and wanting something similar
to Thread::Queue. Check out MCE::Flow and MCE::Queue.
2. Wanting multiple roles and each role uniquely configured with number of
workers and function to execute. Check out flow_demo.pl and step_demo.pl.
3. Wanting to process rows from a database in parallel. Check out sampledb
under the examples directory.
4. MCE can be used to process a single (large) file or process many (small)
files in parallel. Take bin/mce_grep, a wrapper around the grep binary,
for a test drive. MCE also includes egrep.pl for a pure Perl grep
implementation.
5. A large telecom company wanting to poll millions of devices via SNMP.
This is possible with MCE, AnyEvent::SNMP, and Net::SNMP. Think of
having 200 workers and each worker connecting to 300 devices with
AnyEvent. Each worker processes 300 devices simultaneously.
6. Wanting to access many hosts in parallel via SSH or TCP/IP.
...
###############################################################################
# * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * #
###############################################################################
INSTALLATION
To install this module type the following:
MCE_INSTALL_TOOLS=1 perl Makefile.PL (e.g. bin/mce_grep)
(or) perl Makefile.PL
make
make test
make install
DEPENDENCIES
This module requires Perl 5.8.0 or later to run.
MCE spawns child processes by default, not threads.
However, MCE supports threads via 2 threading libraries if threads
is desired.
The use of threads in MCE requires that you include threads support
prior to loading MCE. The use_threads option defaults to 1 when a
thread library is loaded. Threads is loaded automatically for
$^O eq 'MSWin32'.
use threads; use forks;
use threads::shared; (or) use forks::shared;
use MCE; use MCE;
MCE utilizes the following modules:
bytes
constant
Carp
Fcntl
File::Path
IO::Handle
Scalar::Util
Socket
Storable 2.04+
Symbol
Test::More 0.45+ (for testing only via make test)
Time::HiRes
COPYRIGHT AND LICENCE
Copyright (C) 2012-2015 by Mario E. Roy <marioeroy AT gmail DOT com>
This program is free software; you can redistribute it and/or modify it
under the terms of either: the GNU General Public License as published
by the Free Software Foundation; or the Artistic License.
See L<http://dev.perl.org/licenses/> for more information.
USAGE
The source is hosted at http://code.google.com/p/many-core-engine-perl/
https://metacpan.org/module/MCE::Signal
https://metacpan.org/module/MCE
###############################################################################
# * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * #
###############################################################################
Enjoy MCE !!!
- Mario