forked from ritaliu2010/hadoop-hdfs
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathCHANGES.txt
1537 lines (960 loc) · 53.1 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Hadoop HDFS Change Log
Trunk (unreleased changes)
INCOMPATIBLE CHANGES
HDFS-1526. Dfs client name for a map/reduce task should be unique
among threads. (hairong)
HDFS-1536. Improve HDFS WebUI. (hairong)
NEW FEATURES
HDFS-1482. Add listCorruptFileBlocks to DistributedFileSystem.
(Patrick Kling via hairong)
IMPROVEMENTS
HDFS-1510. Added test-patch.properties required by test-patch.sh (nigel)
HDFS-1481. NameNode should validate fsimage before rolling. (hairong)
HDFS-1518. Wrong description in FSNamesystem's javadoc.
(Jingguo Yao via eli)
HDFS-1506. Refactor fsimage loading code. (hairong)
HDFS-1533. A more elegant FileSystem#listCorruptFileBlocks API
(HDFS portion) (Patrick Kling via hairong)
HDFS-1476. listCorruptFileBlocks should be functional while the
name node is in safe mode. (Patrick Kling via hairong)
HDFS-1534. Fix some incorrect logs in FSDirectory. (eli)
HDFS-1539. A config option for the datanode to fsycn a block file
when block is completely written. (dhruba)
HDFS-1547. Improve decommission mechanism. (suresh)
OPTIMIZATIONS
HDFS-1458. Improve checkpoint performance by avoiding unnecessary image
downloads and loading. (hairong)
BUG FIXES
HDFS-1516. mvn-install is broken after 0.22 branch creation. (cos)
HDFS-1360. TestBlockRecovery should bind ephemeral ports.
(Todd Lipcon via hairong)
HDFS-1551. Fix pom templates dependency list (gkesavan)
HDFS-1559. Add missing UGM overrides to TestRefreshUserMappings
(Todd Lipcon via eli)
HDFS-1509. A savenamespace command writes the fsimage and edits into
all configured directories. (dhruba)
HDFS-1540. Make Datanode handle errors from RPC calls to namenode
more elegantly. (dhruba)
HDFS-1463. Accesstime of a file is not updated in safeMode. (dhruba)
Release 0.22.0 - Unreleased
NEW FEATURES
HDFS-992. Re-factor block access token implementation to conform to the
generic Token interface in Common (Kan Zhang and Jitendra Pandey via jghoman)
HDFS-599. Allow NameNode to have a seprate port for service requests from
client requests. (Dmytro Molkov via hairong)
HDFS-1004. Update NN to support Kerberized SSL from HADOOP-6584.
(jghoman and Kan Zhang via jghoman)
HDFS-1005. Fsck security. (borya and Kan Zhang via jghoman)
HDFS-1006. getImage/putImage http requests should be https for the case
of security enabled. (borya and jghoman via jghoman)
HDFS-1033. In secure clusters, NN and SNN should verify that the remote
principal during image and edits transfer. (jghoman)
HDFS-1023. Allow http server to start as regular principal if https
principal not defined. (jghoman)
HDFS-1150. Verify datanodes' identities to clients in secure clusters.
(jghoman)
HDFS-1330. Make RPCs to DataNodes timeout. (hairong)
HDFS-202. HDFS support of listLocatedStatus introduced in HADOOP-6870.
HDFS piggyback block locations to each file status when listing a
directory. (hairong)
HDFS-1361. Add -fileStatus operation to NNThroughputBenchmark. (shv)
HDFS-1435. Provide an option to store fsimage compressed. (hairong)
HDFS-903. Support fsimage validation through MD5 checksum. (hairong)
HDFS-1457. Provide an option to throttle image transmission between
pimary and secondary NameNodes. (Yifei Lu and hairong via hairong)
HDFS-1164. TestHdfsProxy is failing. (Todd Lipcon via cos)
HDFS-811. Add metrics, failure reporting and additional tests for HDFS-457.
(eli)
HDFS-895. Allow hflush/sync to occur in parallel with new writes
to the file. (Todd Lipcon via hairong)
HDFS-528. Add ability for safemode to wait for a minimum number of
live datanodes (Todd Lipcon via eli)
IMPROVEMENTS
HDFS-1304. Add a new unit test for HftpFileSystem.open(..). (szetszwo)
HDFS-1096. fix for prev. commit. (boryas)
HDFS-1096. allow dfsadmin/mradmin refresh of superuser proxy group
mappings (boryas)
HDFS-1146. Javadoc for getDelegationTokenSecretManager in FSNamesystem (jnp via boryas)
HDFS-1132. Refactor TestFileStatus (Eli Collins via cos)
HDFS-1163. normalize property names for JT/NN kerberos principal
names in configuration (from HADOOP 6633) (boryas)
HDFS-1003. authorization checks for inter-server protocol
(based on HADOOP-6600) (boryas)
HDFS-1061. Memory footprint optimization for INodeFile object.
(Bharath Mundlapudi via jghoman)
HDFS-1079. Throw exceptions as specified by the AbstractFileSystem
in HDFS implemenation and protocols. (suresh)
HDFS-1112. Edit log buffer should not grow unfoundedly. (hairong)
HDFS-1119. Introduce a GSet interface to BlocksMap. (szetszwo)
HDFS-1184. Replace tabs in code with spaces. (Jeff Ames via jghoman)
HDFS-1185. Remove duplicate now() functions in DataNode, FSNamesysetm.
(Jeff Ames via jghoman)
HDFS-1183. Remove some duplicate code in NamenodeJspHelper.java.
(Jeff Ames via jghoman)
HDFS-1190. Remove unused getNamenode() method from DataNode.
(Jeff Ames via jghoman)
HDFS-1110. Reuses objects for commonly used file names in namenode to
reduce the heap usage. (suresh)
HDFS-752. Add interfaces classification to to HDFS source code. (suresh)
HDFS-947. An Hftp read request is redirected to a datanode that has
the most replicas of the blocks in the file. (Dmytro Molkov via dhruba)
HDFS-1272. Fixes to take care of the changes in HADOOP-6845.
(Jitendra Pandey via ddas)
HDFS-1298 - Add support in HDFS for new statistics added in FileSystem
to track the file system operations. (suresh)
HDFS-1201. The HDFS component for HADOOP-6632.
(Kan Zhang & Jitendra Pandey via ddas)
HDFS-1307 Add start time, end time and total time taken for FSCK to
FSCK report (suresh)
HDFS-1302. The HDFS side of the changes corresponding to HADOOP-6861.
(Jitendra Pandey & Owen O'Malley via ddas)
HDFS-1315. Add fsck event to audit log and remove other audit log events
corresponding to FSCK listStatus and open calls. (suresh)
HDFS-1178. The NameNode servlets should not use RPC to connect to the
NameNode. (Kan Zhang via jghoman)
HDFS-1130. Adds dfs.cluster.administrator ACL configuration that can
be used to control who can view the default hdfs servlets. (ddas)
HDFS-1297. Fix some comments. (Jeff Ames via jghoman)
HDFS-330. Datanode Web UIs should provide robots.txt.
(Allen Wittenauer via jghoman)
HDFS-881. Refactor DataNode Packet header into DataTransferProtocol.
(Todd Lipcon via jghoman)
HDFS-1036. docs for fetchdt
HDFS-1318. Add JMX interface for read access to namenode and datanode
web UI information. (Tanping Wang via suresh).
HDFS-1356. Provide information as to whether or not security is
enabled on web interface for NameNode (boryas)
HDFS-1205. FSDatasetAsyncDiskService should name its threads.
(Todd Lipcon via eli)
HDFS-1111. Introduce getCorruptFileBlocks() for fsck. (Sriram Rao via shv)
HDFS-1395. Add @Override to FSDataset methods that implement
FSDatasetInterface methods. (suresh)
HDFS-1383. Improve the error messages when using hftp://. (szetszwo)
HDFS-1093. Change the FSNamesystem lock to a read/write lock. (dhruba)
HDFS-1407. Change DataTransferProtocol methods to use Block instead
of individual elements of Block. (suresh)
HDFS-1417. Add @Override to SimulatedFSDataset methods that implement
FSDatasetInterface methods. (suresh)
HDFS-1426. Remove unused method BlockInfo#listCount. (hairong)
HDFS-1456. Provide builder for constructing instances of MiniDFSCluster.
(jghoman)
HDFS-1472. Allow programmatic access to fsck output.
(Ramkumar Vadali via dhruba)
HADOOP-7007. Update the hudson-test-patch ant target to work with the
latest test-patch.sh script (gkesavan)
HDFS-1462. Refactor edit log loading to a separate class from edit log writing.
(Todd Lipcon via eli)
HDFS-1485. Fix typo in BlockPlacementPolicy. (Jingguo Yao via shv)
HDFS-1035. Generate Eclipse's .classpath file from Ivy config. (nigel)
HDFS-1408. Herriot NN and DN clients should vend statistics. (cos)
HDFS-1491 Update Hdfs to match the change of methods from protected to public
in AbstractFileSystem (Hadoop-6903) (sanjay)
HDFS-1160. Improve some FSDataset warnings and comments. (eli)
HDFS-556. Provide info on failed volumes in the web ui. (eli)
HDFS-697. Enable asserts for tests by default. (eli)
HDFS-1187. Modify fetchdt to allow renewing and canceling token.
(Owen O'Malley and Kan Zhang via jghoman)
HDFS-1387. Update HDFS permissions guide for security. (Todd Lipcon via eli)
HDFS-455. Make NN and DN handle in a intuitive way comma-separated
configuration strings. (Michele Catasta via eli)
HDFS-1071. savenamespace should write the fsimage to all configured
fs.name.dir in parallel (Dmytro Molkov via jghoman)
HDFS-1055. Improve thread naming for DataXceivers.
(Todd Lipcon and Ramkumar Vadali via eli).
HDFS-718. Configuration parameter to prevent accidental formatting of
HDFS filesystem. (Andrew Ryan via jghoman)
HDFS-1500. TestOfflineImageViewer failing on trunk. (Todd Lipcon
via hairong)
HDFS-1483. DFSClient.getBlockLocations should indicate if corresponding
blocks are corrupt. (Patrick Kling via hairong)
HDFS-259. Remove intentionally corrupt 0.13 directory layout creation.
(Todd Lipcon via eli)
HDFS-1513. Fix a number of warnings. (eli)
HDFS-1473. Refactor storage management into separate classes than fsimage
file reading/writing. (Todd Lipcon via eli)
OPTIMIZATIONS
HDFS-1140. Speedup INode.getPathComponents. (Dmytro Molkov via shv)
HDFS-1081. Performance regression in
DistributedFileSystem::getFileBlockLocations in secure systems (jghoman)
HDFS-1114. Implement LightWeightGSet for BlocksMap in order to reduce
NameNode memory footprint. (szetszwo)
HDFS-1320. Add LOG.isDebugEnabled() guard for each LOG.debug(..).
(Erik Steffl via szetszwo)
HDFS-1368. Add a block counter to DatanodeDescriptor. (hairong)
HDFS-1434. Refactor Datanode#startDataNode method into smaller methods.
(suresh)
BUG FIXES
HDFS-1039. Adding test for JspHelper.getUGI(jnp via boryas)
HDFS-1019. Incorrect default values for delegation tokens in
hdfs-default.xml (jnp via boryas)
HDFS-1039. Service should be set in the token in JspHelper.getUGI(jnp via boryas)
HDFS-1038. FIX. A test missed in a previous commit for this JIRA. (boryas)
HDFS-1038. In nn_browsedfscontent.jsp fetch delegation token only
if security is enabled. (jnp via boryas)
HDFS-1044. Cannot submit mapreduce job from secure client to
unsecure sever (boryas)
HDFS-1021. specify correct server principal for RefreshAuthorizationPolicyProtocol
and RefreshUserToGroupMappingsProtocol protocols in DFSAdmin (for HADOOP-6612) (boryas)
HDFS-970. fsync fsimage to disk before closing fsimage file.
(Todd Lipcon via dhruba)
HDFS-1027. Update copyright year to 2010. (Ravi Phulari via jghoman)
HDFS-1080. SecondaryNameNode image transfer should use the defined http
address rather than local ip address. (jghoman)
HDFS-1198. Resolving cross-realm principals. (Jitendra Pandey via jghoman)
HDFS-1118. Fix socketleak on DFSClient. (Zheng Shao via dhruba)
HDFS-1192. refreshSuperUserGroupsConfiguration should use server side
configuration for the refresh (for HADOOP-6815) (boryas)
HDFS-1036. in DelegationTokenFetch dfs.getURI returns no port (boryas)
HDFS-1017. browsedfs jsp should call JspHelper.getUGI rather
than using createRemoteUser() (jnp via boryas)
HDFS-1250. Namenode should reject block reports and block received
requests from dead datanodes (suresh)
HDFS-1145. When NameNode is shutdown it does not try to exit
safemode anymore. (dhruba)
HDFS-1202. DataBlockScanner throws NPE when updated before
initialized. (Todd Lipcon via dhruba)
HDFS-882. Datanode logs the hostname and port its listening on.
(Steve Loughran via dhruba)
HDFS-1238. ant eclipse-files has drifted again, (jghoman)
HDFS-1045. In secure clusters, re-login is necessary for https
clients before opening connections. (jghoman)
HDFS-1289. Datanode secure mode is broken. (Kan Zhang via jghoman)
HDFS-1007. HFTP needs to be updated to use delegation tokens (boryas)
HDFS-1085. HFTP read may fail silently on the client side if there is an
exception on the server side. (szetszwo)
HDFS-1308. job conf key for the services name of DelegationToken for HFTP
url is constructed incorrectly in HFTPFileSystem (boryas)
HDFS-1319. Fix location of re-login for secondary namenode from HDFS-999.
(jghoman)
HDFS-1317. Remove the FILEPATH_PATTERN from hdfsproxy.AuthorizationFilter.
(Rohini Palaniswamy via szetszwo)
HDFS-912. sed in build.xml on Solaris fails. (Allen Wittenauer via jghoman)
HDFS-1296. using delegation token over hftp for long running
clients (boryas)
HDFS-1334. open in HftpFileSystem does not add delegation tokens to the url.
(Jitendra Pandey via jghoman)
HDFS-1301. TestHDFSProxy need to use server side conf for ProxyUser
stuff. (boryas)
HDFS-1340. When security is turned off, there is a potential XSS attack.
This patch fixes it by removing delegationtoken string from the URL,
before returning a response to the client. (Jitendra Pandey via ddas)
HDFS-1347. TestDelegationToken uses mortbay.log for logging (boryas)
HDFS-1157. Modifications introduced by HDFS-1150 are breaking aspect's
bindings (cos)
HDFS-1349. Remove empty java files. (Eli Collins)
HDFS-1340. A null delegation token is appended to the url if security
is disabled when browsing filesystem. (boryas)
HDFS-1352. Fix jsvc.location. (Eli Collins via jghoman)
HDFS-1284. TestBlockToken fails. (Kan Zhang via jghoman)
HDFS-1355. ant veryclean (clean-cache) doesn't clean enough.
(Luke Lu via jghoman)
HDFS-1353. Remove most of getBlockLocation optimization. (jghoman)
HDFS-1369. Invalid javadoc reference in FSDatasetMBean.java (Eli Collins)
HDFS-829. hdfsJniHelper.c: #include <error.h> is not portable.
(Allen Wittenauer via jghoman)
HDFS-1310. The ClientDatanodeProtocol proxy should be stopped in
DFSInputStream.readBlockLength(..). (sam rash via szetszwo)
HDFS-1357. HFTP traffic served by DataNode shouldn't use service port
on NameNode. (Kan Zhang via jghoman)
HDFS-96. HDFS supports blocks larger than 2 GB.
(Patrick Kling via dhruba)
HDFS-1433. Fix test failures - TestPread and TestFileLimit. (suresh)
HDFS-1364. Makes long running HFTP-based applications do relogins
if necessary. (Jitendra Pandey via ddas)
HDFS-1399. Distinct minicluster services (e.g. NN and JT) overwrite each
other's service policies. (Aaron T. Myers via tomwhite)
HDFS-1440. Fix TestComputeInvalidateWork failure. (suresh)
HDFS-1498. FSDirectory#unprotectedConcat calls setModificationTime
on a file. (eli)
HDFS-1466. TestFcHdfsSymlink relies on /tmp/test not existing. (eli)
HDFS-874. TestHDFSFileContextMainOperations fails on weirdly
configured DNS hosts. (Todd Lipcon via eli)
HDFS-1507. TestAbandonBlock should abandon a block. (eli)
HDFS-1487. FSDirectory.removeBlock() should update diskspace count
of the block owner node (Zhong Wang via eli).
HDFS-1001. DataXceiver and BlockReader disagree on when to send/recv
CHECKSUM_OK. (bc Wong via eli)
HDFS-1167. New property for local conf directory in system-test-hdfs.xml
file. (Vinay Thota via cos)
HDFS-1467. Append pipeline never succeeds with more than one replica.
(Todd Lipcon via eli)
HDFS-1503. TestSaveNamespace fails. (Todd Lipcon via cos)
HDFS-1524. Image loader should make sure to read every byte in image file.
(hairong)
HDFS-1523. TestLargeBlock is failing on trunk. (cos)
HDFS-1502. TestBlockRecovery triggers NPE in assert. (hairong via cos)
HDFS-1532. Exclude Findbugs warning in FSImageFormat$Saver. (Todd Lipcon
via cos)
HDFS-1527. SocketOutputStream.transferToFully fails for blocks >= 2GB on
32 bit JVM. (Patrick Kling via cos)
HDFS-1531. Clean up stack traces due to duplicate MXBean registration.
(Todd Lipcon via cos)
HDFS-613. TestBalancer and TestBlockTokenWithDFS fail Balancer assert.
(Todd Lipcon via cos)
HDFS-1511. 98 Release Audit warnings on trunk and branch-0.22.
(jghoman)
HDFS-1560. dfs.data.dir permissions should default to 700.
(Todd Lipcon via eli)
HDFS-1550. NPE when listing a file with no location. (hairong)
HDFS-1542. Add test for HADOOP-7082, a deadlock writing Configuration to
HDFS. (todd)
HDFS-1504. FSImageSaver should catch all exceptions, not just IOE. (todd)
HDFS-884. DataNode throws IOException if all data directories are
unavailable. (Steve Loughran and shv)
HDFS-1572. Checkpointer should trigger checkpoint with specified period.
(jghoman)
Release 0.21.1 - Unreleased
IMPROVEMENTS
HDFS-1411. Correct backup node startup command in hdfs user guide.
(Ching-Shen Chen via shv)
BUG FIXES
HDFS-1363. Eliminate second synchronized sections in appendFile(). (shv)
HDFS-1413. Fix broken links to HDFS Wiki. (shv)
HDFS-1420. Clover build doesn't generate per-test coverage (cos)
HDFS-1444. Test related code of build.xml is error-prone and needs to be
re-aligned. (cos)
HDFS-1343. Instrumented build should be concentrated in one build area (cos)
HDFS-1452. ant compile-contrib is broken (cos)
HDFS-1474. ant binary-system is broken (cos)
HDFS-1292. Allow artifacts to be published to the staging Apache Nexus
Maven Repository. (Giridharan Kesavan via tomwhite)
HDFS-1206. TestFiHFlush fails intermittently. (cos)
HDFS-1548. Fault-injection tests are executed multiple times if invoked
with run-test-hdfs-fault-inject target (cos)
HDFS-1552. Remove java5 dependencies from build. (cos)
HDFS-996. JUnit tests should never depend on anything in conf (cos)
INCOMPATIBLE CHANGES
HDFS-538. Per the contract elucidated in HADOOP-6201, throw
FileNotFoundException from FileSystem::listStatus rather than returning
null. (Jakob Homan via cdouglas)
HDFS-602. DistributedFileSystem mkdirs throws FileAlreadyExistsException
instead of FileNotFoundException. (Boris Shkolnik via suresh)
HDFS-544. Add a "rbw" subdir to DataNode data directory. (hairong)
HDFS-576. Block report includes under-construction replicas. (shv)
HDFS-636. SafeMode counts complete blocks only. (shv)
HDFS-644. Lease recovery, concurrency support. (shv)
HDFS-570. Get last block length from a data-node when opening a file
being written to. (Tsz Wo (Nicholas), SZE via shv)
HDFS-657. Remove unused legacy data-node protocol methods. (shv)
HDFS-658. Block recovery for primary data-node. (shv)
HDFS-660. Remove deprecated methods from InterDatanodeProtocol. (shv)
HDFS-512. Block.equals() and compareTo() compare blocks based
only on block Ids, ignoring generation stamps. (shv)
HDFS-873. Configuration specifies data-node storage directories as URIs.
(shv)
HDFS-905. Use the new UserGroupInformation from HDFS-6299.
(jghoman via omalley)
HDFS-984. Persistent delegation tokens. (Jitendra Pandey via shv)
HDFS-1016. HDFS side change for HADOOP-6569. This jira changes the
error message on the screen when cat a directory or a
non-existent file. (hairong)
NEW FEATURES
HDFS-1134. Large-scale Automated Framework. (cos)
HDFS-436. Introduce AspectJ framework for HDFS code and tests.
(Konstantin Boudnik via szetszwo)
HDFS-447. Add LDAP lookup to hdfsproxy. (Zhiyong Zhang via cdouglas)
HDFS-459. Introduce Job History Log Analyzer. (shv)
HDFS-461. Tool to analyze file size distribution in HDFS. (shv)
HDFS-492. Add two JSON JSP pages to the Namenode for providing corrupt
blocks/replicas information. (Bill Zeller via szetszwo)
HDFS-578. Add support for new FileSystem method for clients to get server
defaults. (Kan Zhang via suresh)
HDFS-595. umask settings in configuration may now use octal or symbolic
instead of decimal. (Jakob Homan via suresh)
HADOOP-6234. Updated hadoop-core and test jars to propagate new option
dfs.umaskmode in configuration. (Jakob Homan via suresh)
HDFS-235. Add support for byte ranges in HftpFileSystem to serve
range of bytes from a file. (Bill Zeller via suresh)
HDFS-385. Add support for an experimental API that allows a module external
to HDFS to specify how HDFS blocks should be placed. (dhruba)
HADOOP-4952. Update hadoop-core and test jars to propagate new FileContext
file system application interface. (Sanjay Radia via suresh).
HDFS-567. Add block forensics contrib tool to print history of corrupt and
missing blocks from the HDFS logs.
(Bill Zeller, Jitendra Nath Pandey via suresh).
HDFS-610. Support o.a.h.fs.FileContext. (Sanjay Radia via szetszwo)
HDFS-536. Support hflush at DFSClient. (hairong)
HDFS-517. Introduce BlockInfoUnderConstruction to reflect block replica
states while writing. (shv)
HDFS-565. Introduce block committing logic during new block allocation
and file close. (shv)
HDFS-537. DataNode exposes a replica's meta info to BlockReceiver for the
support of dfs writes/hflush. It also updates a replica's bytes received,
bytes on disk, and bytes acked after receiving a packet. (hairong)
HDFS-585. Datanode should serve up to visible length of a replica for read
requests. (szetszwo)
HDFS-604. Block report processing for append. (shv)
HDFS-619. Support replica recovery initialization in datanode for the new
append design. (szetszwo)
HDFS-592. Allow clients to fetch a new generation stamp from NameNode for
pipeline recovery. (hairong)
HDFS-624. Support a new algorithm for pipeline recovery and pipeline setup
for append. (hairong)
HDFS-627. Support replica update in data-node.
(Tsz Wo (Nicholas), SZE and Hairong Kuang via shv)
HDFS-642. Support pipeline close and close error recovery. (hairong)
HDFS-631. Rename configuration keys towards API standardization and
backward compatibility. (Jitendra Nath Pandey via suresh)
HDFS-669. Add unit tests framework (Mockito) (cos, Eli Collins)
HDFS-731. Support new Syncable interface in HDFS. (hairong)
HDFS-702. Add HDFS implementation of AbstractFileSystem.
(Sanjay Radio via suresh)
HDFS-758. Add decommissioning status page to Namenode Web UI.
(Jitendra Nath Pandey via suresh)
HDFS-814. Add an api to get the visible length of a DFSDataInputStream.
(szetszwo)
HDFS-654. Add support new atomic rename functionality in HDFS for
supporting rename in FileContext. (suresh)
HDFS-222. Support for concatenating of files into a single file
without copying. (Boris Shkolnik via hairong)
HDFS-933. Adds Delegation token based authentication in the NameNode.
(Kan Zhang via ddas)
HDFS-935. Adds a real user component in Delegation token.
(Jitendra Nath Pandey via ddas)
HDFS-245. Adds a symlink implementation to HDFS. This complements the new
symlink feature added in HADOOP-6421 (Eli Collins via Sanjay Radia)
HDFS-1009. Support Kerberos authorization in HDFSProxy. (Srikanth
Sundarrajan via szetszwo)
HDFS-1091. Implement listStatus that returns an iterator of FileStatus.
(hairong)
IMPROVEMENTS
HDFS-381. Remove blocks from DataNode maps when corresponding file
is deleted. (Suresh Srinivas via rangadi)
HDFS-377. Separate codes which implement DataTransferProtocol.
(szetszwo)
HDFS-396. NameNode image and edits directories are specified as URIs.
(Luca Telloli via rangadi)
HDFS-444. Allow to change probability levels dynamically in the fault
injection framework. (Konstantin Boudnik via szetszwo)
HDFS-352. Documentation for saveNamespace command. (Ravi Phulari via shv)
HADOOP-6106. Updated hadoop-core and test jars from hudson trunk
build #12. (Giridharan Kesavan)
HDFS-204. Add a new metrics FilesInGetListingOps to the Namenode.
(Jitendra Nath Pandey via szetszwo)
HDFS-278. HDFS Outputstream close does not hang forever. (dhruba)
HDFS-443. Add a new metrics numExpiredHeartbeats to the Namenode.
(Jitendra Nath Pandey via szetszwo)
HDFS-475. Add new ant targets for fault injection jars and tests.
(Konstantin Boudnik via szetszwo)
HDFS-458. Create a new ant target, run-commit-test. (Jakob Homan
via szetszwo)
HDFS-493. Change build.xml so that the fault-injected tests are executed
only by the run-test-*-fault-inject targets. (Konstantin Boudnik via
szetszwo)
HDFS-446. Improvements to Offline Image Viewer. (Jakob Homan via shv)
HADOOP-6160. Fix releaseaudit target to run on specific directories.
(gkesavan)
HDFS-501. Use enum to define the constants in DataTransferProtocol.
(szetszwo)
HDFS-508. Factor out BlockInfo from BlocksMap. (shv)
HDFS-510. Rename DatanodeBlockInfo to be ReplicaInfo.
(Jakob Homan & Hairong Kuang via shv)
HDFS-500. Deprecate NameNode methods deprecated in NameNodeProtocol.
(Jakob Homan via shv)
HDFS-514. Change DFSClient.namenode from public to private. (Bill Zeller
via szetszwo)
HDFS-496. Use PureJavaCrc32 in HDFS. (Todd Lipcon via szetszwo)
HDFS-511. Remove redundant block searches in BlockManager. (shv)
HDFS-504. Update the modification time of a file when the file
is closed. (Chun Zhang via dhruba)
HDFS-498. Add development guide and documentation for the fault injection
framework. (Konstantin Boudnik via szetszwo)
HDFS-524. Further DataTransferProtocol code refactoring. (szetszwo)
HDFS-529. Use BlockInfo instead of Block to avoid redundant block searches
in BlockManager. (shv)
HDFS-530. Refactor TestFileAppend* to remove code duplication.
(Konstantin Boudnik via szetszwo)
HDFS-451. Add fault injection tests for DataTransferProtocol. (szetszwo)
HDFS-409. Add more access token tests. (Kan Zhang via szetszwo)
HDFS-546. DatanodeDescriptor iterates blocks as BlockInfo. (shv)
HDFS-457. Do not shutdown datanode if some, but not all, volumes fail.
(Boris Shkolnik via szetszwo)
HDFS-548. TestFsck takes nearly 10 minutes to run. (hairong)
HDFS-539. Refactor fault injeciton pipeline test util for future reuse.
(Konstantin Boudnik via szetszwo)
HDFS-552. Change TestFiDataTransferProtocol to junit 4 and add a few new
tests. (szetszwo)
HDFS-563. Simplify the codes in FSNamesystem.getBlockLocations(..).
(szetszwo)
HDFS-581. Introduce an iterator over blocks in the block report array.(shv)
HDFS-549. Add a new target, run-with-fault-inject-testcaseonly, which
allows an execution of non-FI tests in FI-enable environment. (Konstantin
Boudnik via szetszwo)
HDFS-173. Namenode will not block until a large directory deletion
completes. It allows other operations when the deletion is in progress.
(suresh)
HDFS-551. Create new functional test for a block report. (Konstantin
Boudnik via hairong)
HDFS-288. Redundant computation in hashCode() implementation.
(szetszwo via tomwhite)
HDFS-412. Hadoop JMX usage makes Nagios monitoring impossible.
(Brian Bockelman via tomwhite)
HDFS-472. Update hdfsproxy documentation. Adds a setup guide and design
document. (Zhiyong Zhang via cdouglas)
HDFS-617. Support non-recursive create(). (Kan Zhang via szetszwo)
HDFS-618. Support non-recursive mkdir(). (Kan Zhang via szetszwo)
HDFS-574. Split the documentation between the subprojects.
(Corinne Chandel via omalley)
HDFS-598. Eclipse launch task for HDFS. (Eli Collins via tomwhite)
HDFS-641. Move all of the components that depend on map/reduce to
map/reduce. (omalley)
HDFS-509. Redesign DataNode volumeMap to include all types of Replicas.
(hairong)
HDFS-562. Add a test for NameNode.getBlockLocations(..) to check read from
un-closed file. (szetszwo)
HDFS-543. Break FSDatasetInterface#writToBlock() into writeToRemporary,
writeToRBW, ad append. (hairong)
HDFS-603. Add a new interface, Replica, which is going to replace the use
of Block in datanode. (szetszwo)
HDFS-589. Change block write protocol to support pipeline recovery.
(hairong)
HDFS-652. Replace BlockInfo.isUnderConstruction() with isComplete() (shv)
HDFS-648. Change some methods in AppendTestUtil to public. (Konstantin
Boudnik via szetszwo)
HDFS-662. Unnecessary info message from DFSClient. (hairong)
HDFS-518. Create new tests for Append's hflush. (Konstantin Boudnik
via szetszwo)
HDFS-688. Add configuration resources to DFSAdmin. (shv)
HDFS-29. Validate the consistency of the lengths of replica and its file
in replica recovery. (szetszwo)
HDFS-680. Add new access method to a copy of a block's replica. (shv)
HDFS-704. Unify build property names to facilitate cross-projects
modifications (cos)
HDFS-705. Create an adapter to access some of package-private methods of
DataNode from tests (cos)
HDFS-710. Add actions with constraints to the pipeline fault injection
tests and change SleepAction to support uniform random sleeping over an
interval. (szetszwo)
HDFS-713. Need to properly check the type of the test class from an aspect
(cos)
HDFS-716. Define a pointcut for pipeline close and add a few fault
injection tests to simulate out of memory problem. (szetszwo)
HDFS-719. Add 6 fault injection tests for pipeline close to simulate slow
datanodes and disk errors. (szetszwo)
HDFS-616. Create functional tests for new design of the block report. (cos)
HDFS-584. Fail the fault-inject build if any advices are mis-bound. (cos)
HDFS-730. Add 4 fault injection tests to simulate non-responsive datanode
and out-of-memory problem for pipeline close ack. (szetszwo)
HDFS-728. Create a comprehensive functional test for append. (hairong)
HDFS-736. commitBlockSynchronization() updates block GS and length
in-place. (shv)
HADOOP-5107. Use Maven ant tasks to publish the subproject jars.
(Giridharan Kesavan via omalley)
HDFS-521. Create new tests for pipeline (cos)
HDFS-764. Places the Block Access token implementation in hdfs project.
(Kan Zhang via ddas)
HDFS-787. Upgrade some libraries to be consistent with common and
mapreduce. (omalley)
HDFS-519. Create new tests for lease recovery (cos)
HDFS-804. New unit tests for concurrent lease recovery (cos)
HDFS-813. Enable the append test in TestReadWhileWriting. (szetszwo)
HDFS-145. Cleanup inconsistent block length handling code in
FSNameSystem#addStoredBlock. (hairong)
HDFS-127. Reset failure count in DFSClient for each block acquiring
operation. (Igor Bolotin via szetszwo)
HDFS-520. Create new tests for block recovery. (hairong)
HDFS-1067. Create block recovery tests that handle errors. (hairong)
HDFS-1107. Turn on append by default. (shv)
HDFS-968. Use StringBuilder instead of StringBuffer for better
performance. (Kay Kay via suresh)
HDFS-703. Replace current fault injection implementation with one
from (cos)
HDFS-754. Reduce ivy console output to observable level (cos)
HDFS-832. HDFS side of HADOOP-6222. (cos)
HDFS-840. Change tests to use FileContext test helper introduced in
HADOOP-6394. (Jitendra Nath Pandey via suresh)
HDFS-685. Use the user-to-groups mapping service in the NameNode.
(boryas, acmurthy)
HDFS-755. Read multiple checksum chunks at once in DFSInputStream.
(Todd Lipcon via tomwhite)
HDFS-786. Implement getContentSummary in HftpFileSystem.
(Tsz Wo (Nicholas), SZE via cdouglas)
HDFS-587. Add support for specifying queue name in mapreduce tests.
(Erik Steffl via suresh)
HDFS-902 Move contrib/raid to MapReduce. (Eli Collins via omalley)
HDFS-800. The last block of a file under construction may change to the
COMPLETE state in response to getAdditionalBlock or completeFileInternal.
(hairong)
HDFS-899. Delegation Token Implementation
and corresponding changes in Namenode and DFS Api to issue,
renew and cancel delegation tokens. (jnp via boryas)
HDFS-844. Log the filename when file locking fails. (tomwhite)
HDFS-914. Refactor DFSOutputStream and DFSInputStream out of DFSClient.
(Todd Lipcon via tomwhite)
HDFS-949. Move DelegationToken into Common so that it can be used by
MapReduce. (omalley)
HDFS-930. Better error message for DATA_TRANSFER_VERSION mismatched.
(Kay Kay via szetszwo)
HDFS-986. Delegation token renewing and cancelling should provide
meaningful exceptions when there are failures instead of returning
false. (omalley)
HADOOP-6579. Upgrade the commons-codec library to 1.4. (omalley)
HDFS-991. Allow authentication to the web ui via a delegation token.
(omalley)
HDFS-994. Allow fetching of delegation token from NameNode for hftp.
(Jakob Homan via acmurthy)
HDFS-998. Quote blocks streamed through jsps. (cdouglas)
HDFS-729. NameNode API to list files that have missing blocks.
(Rodrigo Schmidt via dhruba)
HDFS-850. The WebUI display more details about namenode memory usage.
(Dmytro Molkov via dhruba)
HDFS-826. The DFSOutputStream has a API that returns the number of
active datanode(s) in the current pipeline. (dhruba)
HDFS-985. HDFS should issue multiple RPCs for listing a large
directory. (hairong)
HDFS-1043. NNThroughputBenchmark modifications to support benchmarking of
server-side user group resolution. (shv)
HDFS-892. Optionally use Avro reflection for Namenode RPC. This
is not a complete implementation yet, but rather a starting point.
(cutting)
HDFS-854. Datanode should scan devices in parallel to generate
block report. (Dmytro Molkov via jhoman)
HDFS-1032. fsck has an option to list corrupt files.
(Andre Oriai via dhruba)
HDFS-1024. SecondaryNameNode verifies size of fsimage and edits file.
(Dmytro Molkov via dhruba)
HDFS-1011. hdfsproxy: Improve log messages by restoring the previous
thread name. (Srikanth Sundarrajan via szetszwo)
HDFS-997. Allow datanode storage directory permissions to be configurable.
(Luke Lu via cdouglas)
HDFS-1012. hdfsproxy: Support for fully qualified HDFS path in addition to
simple unqualified path. (Srikanth Sundarrajan via szetszwo)
HDFS-933. Namenode should issue a delegation token only for kerberos
authenticated clients.(jnp via boryas)