From 15e93ef2d591700a86fceb7e0d23d0cd6c1b4683 Mon Sep 17 00:00:00 2001 From: Nikitha Chettiar Date: Sun, 7 Feb 2021 02:43:21 -0800 Subject: [PATCH 1/3] Adding design folder --- design/large-file-upload-task-design.md | 73 +++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 design/large-file-upload-task-design.md diff --git a/design/large-file-upload-task-design.md b/design/large-file-upload-task-design.md new file mode 100644 index 000000000..9ef06612a --- /dev/null +++ b/design/large-file-upload-task-design.md @@ -0,0 +1,73 @@ +##### LargeFileUpload Task Design Changes + +This document proposes high-level design modifications to the `LargeFileUploadTask` implementation. Reasons for change - + +- Enhancement - Support Node.js Stream upload. Issue [#320](https://github.com/microsoftgraph/msgraph-sdk-javascript/issues/320). +- Bug Fix - Support large file uploads to Outlook API. Issue [#359](https://github.com/microsoftgraph/msgraph-sdk-javascript/issues/359). + +Outline of the current implementation - + +``` +interface FileObject { + content: ArrayBuffer | File; + name: string; + size: number; +} + +// Create a LargeFileUploadTask object with the file object +constructor(client: Client, file: FileObject, uploadSession: LargeFileUploadSession, options: LargeFileUploadTaskOptions = {}) + +// Call the LargeFileUploadTask.upload() function to upload large file to the API in ranges. +upload() { + while (true) { + const fileSlice = this.sliceFile(nextRange); + const response = await this.uploadSlice(fileSlice, nextRange, this.file.size); + // Upon completion of upload process incase of onedrive, driveItem is returned, which contains id + if (response.id !== undefined) { + return response; + } else { + this.updateTaskStatus(response); + } + } +} + +// Function to slice the file as per the "Next Expected Ranges". +sliceFile(range: Range): ArrayBuffer | Blob { + const blob = this.file.content.slice(range.minValue, range.maxValue + 1); + return blob; +} +``` + +###### 1. Support Node.js Stream upload + +- The requirement is to allow a `FileObject` with content of type [Readable Stream](https://nodejs.org/api/stream.html#stream_class_stream_readable) +- The `file.slice()` function is not applicable to `Readable Stream` and logic for splitting and handling `Stream` varies. +- Note - Chunk uploading is not supported by the Graph API and the stream will be split into multiple ranges and each range will uploaded sequentially. + +- Proposed changes in the current design - + - Move `sliceFlice()` in the FileObject Interface + ``` + interface FileObject{ + sliceFile(range: Range): ArrayBuffer | Blob + } + ``` + - Introduce new classes implementing the `FileObject` interface. Example - `StreamLargeFile.ts` will contain the `sliceFile` function and logic to handle stream classes. + - Change in the upload function will look like + ``` + upload() { + // current : const fileSlice = this.sliceFile(nextRange); + // proposed change is as follows - + const fileSlice = this.fileObject.sliceFile(Range); + } + ``` + - This change is based on idea of Dependency Inversion principle. The goal is to depend on abstractions so that we can easily add or modify support for different file formats in the future or allow customized `FileObject` implementations. +- For browser support, take a dependency on [stream-browserify](https://www.npmjs.com/package/stream-browserify) - the stream module from node core, for browsers and make necessary updates to the rollup or bundling process. + +###### 2. Support large file uploads to Outlook API + +- Bug in the current implemenation - `response.id !== undefined` this condition to mark the completion of an upload does not work for Outlook API since the final response does not contain a response body. +- An upload task should be marked as completed if the response status is a 201. [SDK-design document](https://github.com/microsoftgraph/msgraph-sdk-design/blob/master/tasks/FileUploadTask.md). +- The LargeFileUploadTask should allow uploads to OneDrive API, Outlook API and PrintDocument API. +- Proposed changes- + - Add class `OutlookLargeFileUploadTask.ts` extending the `LargeFileUploadTask` similar to `OneDriveLargeFileUploadTask.ts`. This allows to handle Outlook API specific file upload customizations. + - Currently the `upload` function in the `LargeFileUploadTask` returns only the response body. This can be changed to return the raw response received from the API which can filtered in the child class implementations. From 4a1d45b83e07e440ecbc43384cf123a78e03b4dc Mon Sep 17 00:00:00 2001 From: Nikitha Chettiar Date: Tue, 16 Feb 2021 23:31:33 -0800 Subject: [PATCH 2/3] Adding upload progress design plans --- design/large-file-upload-task-design.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/design/large-file-upload-task-design.md b/design/large-file-upload-task-design.md index 9ef06612a..45e1bfb17 100644 --- a/design/large-file-upload-task-design.md +++ b/design/large-file-upload-task-design.md @@ -4,10 +4,15 @@ This document proposes high-level design modifications to the `LargeFileUploadTa - Enhancement - Support Node.js Stream upload. Issue [#320](https://github.com/microsoftgraph/msgraph-sdk-javascript/issues/320). - Bug Fix - Support large file uploads to Outlook API. Issue [#359](https://github.com/microsoftgraph/msgraph-sdk-javascript/issues/359). +- Enhancement- Support upload progress handler callback. Issue [#305](https://github.com/microsoftgraph/msgraph-sdk-javascript/issues/305). Outline of the current implementation - ``` +interface LargeFileUploadTaskOptions { + rangeSize?: number; +} + interface FileObject { content: ArrayBuffer | File; name: string; @@ -71,3 +76,17 @@ sliceFile(range: Range): ArrayBuffer | Blob { - Proposed changes- - Add class `OutlookLargeFileUploadTask.ts` extending the `LargeFileUploadTask` similar to `OneDriveLargeFileUploadTask.ts`. This allows to handle Outlook API specific file upload customizations. - Currently the `upload` function in the `LargeFileUploadTask` returns only the response body. This can be changed to return the raw response received from the API which can filtered in the child class implementations. + +###### 3. Support upload progress handler callback +- Proposed changes - + - Add interface -> `interface Progress{ + progress(range: Range):void + }` + - Add progressCallBack option to -> + ``` + interface LargeFileUploadTaskOptions { + rangeSize?: number; + progressCallBack?: Progress; + } + ``` + - In the `upload` function call the `progressCallBack.progress()` function if defined. From b85a52b4fc2ebedb5dd53a698ee2bc22385e99f0 Mon Sep 17 00:00:00 2001 From: Nikitha Chettiar Date: Tue, 20 Apr 2021 12:07:29 -0700 Subject: [PATCH 3/3] Adding confirmed changes --- design/large-file-upload-task-design.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/design/large-file-upload-task-design.md b/design/large-file-upload-task-design.md index 45e1bfb17..cf4b3468b 100644 --- a/design/large-file-upload-task-design.md +++ b/design/large-file-upload-task-design.md @@ -8,7 +8,7 @@ This document proposes high-level design modifications to the `LargeFileUploadTa Outline of the current implementation - -``` +```TypeScript interface LargeFileUploadTaskOptions { rangeSize?: number; } @@ -51,7 +51,7 @@ sliceFile(range: Range): ArrayBuffer | Blob { - Proposed changes in the current design - - Move `sliceFlice()` in the FileObject Interface - ``` + ```TypeScript interface FileObject{ sliceFile(range: Range): ArrayBuffer | Blob } @@ -74,19 +74,22 @@ sliceFile(range: Range): ArrayBuffer | Blob { - An upload task should be marked as completed if the response status is a 201. [SDK-design document](https://github.com/microsoftgraph/msgraph-sdk-design/blob/master/tasks/FileUploadTask.md). - The LargeFileUploadTask should allow uploads to OneDrive API, Outlook API and PrintDocument API. - Proposed changes- - - Add class `OutlookLargeFileUploadTask.ts` extending the `LargeFileUploadTask` similar to `OneDriveLargeFileUploadTask.ts`. This allows to handle Outlook API specific file upload customizations. - - Currently the `upload` function in the `LargeFileUploadTask` returns only the response body. This can be changed to return the raw response received from the API which can filtered in the child class implementations. + - Add class `UploadResult` containing `location` and `responseBody` properties. + - `location` provides access to the `location` field in the response headers. + - `responseBody` provides access to the Graph API response body. + - The `upload` task should return the `UploadResult` object on successful completion of task. ###### 3. Support upload progress handler callback - Proposed changes - - - Add interface -> `interface Progress{ - progress(range: Range):void + - Add interface -> `interface UploadEventHandler{ + extraCallbackParam?: unknown; + progress(range: Range, extraCallbackParam?: unknown):void }` - - Add progressCallBack option to -> + - Add uploadEventHandlers option to -> ``` interface LargeFileUploadTaskOptions { rangeSize?: number; - progressCallBack?: Progress; + uploadEventHandlers?: UploadEventHandler; } ``` - - In the `upload` function call the `progressCallBack.progress()` function if defined. + - In the `upload` function call the `uploadEventHandlers.progress()` function if defined.