Blob storage support for large files with no overhead #463

14 Answer(s)

• 

  0

  alper created 3 years ago

  Support Team Director

  not everyone is dealing with raw access to streams. but there's nothing that prevents you to create a new controller and transfer raw streams. by the way I've created an internal issue for this topic. the framework team will make these enhancements.

  Save Cancel
• 

  0

  alexandru-bagu created 3 years ago

  Nothing prevents me to create a new controller, that is true but there is no support/documentation for how to authenticate a http client to be able to use the authorization policies already in place.

  Save Cancel
• 

  0

  alper created 3 years ago

  Support Team Director

  maybe this works for you https://github.com/abpframework/abp/blob/dev/framework/src/Volo.Abp.Cli.Core/Volo/Abp/Cli/Http/CliHttpClient.cs

  This is the code that ABP CLI tool authenticates to the abp.io which is made on top of ABP Framework.

  Save Cancel
• 

  0

  alper created 3 years ago

  Support Team Director

  also see this code, this is used to upload a file which is written in a controller

  [RemoteService(Name = FileManagementRemoteServiceConsts.RemoteServiceName)]
  [Area("fileManagement")]
  [ControllerName("FileDescriptors")]
  [Route("api/file-management/file-descriptor")]
  [Authorize(FileManagementPermissions.FileDescriptor.Default)]
  public class FileDescriptorController : AbpController, IFileDescriptorAppService
  {	
  	[HttpPost]
  	[Route("upload")]
  	[Authorize(FileManagementPermissions.FileDescriptor.Create)]
  	[ApiExplorerSettings(IgnoreApi = true)]
  	public virtual async Task<IActionResult> UploadAsync(Guid? directoryId, IFormFile file)
  	{
  		if (file == null)
  		{
  			return BadRequest();
  		}
  
  		using (var memoryStream = new MemoryStream())
  		{
  			await file.CopyToAsync(memoryStream);
  
  			var fileDescriptor = await FileDescriptorAppService.CreateAsync(
  				new CreateFileInput
  				{
  					Name = file.FileName,
  					MimeType = file.ContentType,
  					DirectoryId = directoryId,
  					Content = memoryStream.ToArray()
  				}
  			);
  
  			return StatusCode(201, fileDescriptor);
  		}
  	}
  }
  

  Save Cancel
• 

  0

  alexandru-bagu created 3 years ago

  Totally unhelpful.

  https://github.com/abpframework/abp/blob/dev/framework/src/Volo.Abp.Cli.Core/Volo/Abp/Cli/Http/CliHttpClient.cs does not have any authentication provided to it.

  The code for the FileDescriptorController uses IFormFile which is buffered according to MSDN. More over, based on default settings for IIS, buffered requests cannot go past 30 MB.

  Save Cancel
• 

  0

  alper created 3 years ago

  Support Team Director

  see this https://stackoverflow.com/a/42460443/1767482 also application services shouldn't use IFormFile or Stream, use byte[] for files in your application services.

  Save Cancel
• 

  0

  alexandru-bagu created 3 years ago

  Say what now? So I should waste all of my server's memory just because "application services shouldn't use IFormFile or Stream"? I agree that IFormFile should only be in the api controller but stream? What you are probably trying to say is that because most of the time application services interfaces are accessed using proxies one must use byte[] because Stream is not serializable. However that does not show that the proper way to handle this kind of work is using byte[], That just shows that you don't have any proper support for streams.

  As this conversation continues it seems I need to clarify some things. I already have full support of streams done in my code, that is not the point of this thread. My question is whether any proper support for streaming will be included in ABP and whether there will be any documentation on how to authenticate an HttpClient inside a scope that already has application services proxies working.

  At the moment there is no proper way to do any of this other than avoiding app service proxies and digging through abp framework's code to see what one has to copy from there to actually be able to create his own HttpClient and authenticate it to be able to do custom requests.

  Save Cancel
• 

  0

  alper created 3 years ago

  Support Team Director

  I'm trying to find a quick solution as there's no support for streaming from application service. The issue must be solved in the framework. You can track this issue https://github.com/abpframework/abp/issues/5727

  Save Cancel
• 

  0

  alexandru-bagu created 3 years ago

  For anyone who is interested in a solution using the current ABP version (3.2.0) you must do all the hardwork yourself. While this issue has been assigned to version 4.1 preview, I took it upon myself to implement it and I've already created a pull request for it.

  If you want to do this using the current version you will need a few things:

  1. IStreamService<TService> in <Namespace>.HttpApi.Client

     public interface IStreamService<TService>
      {
          Task UploadAsync(string route, Stream stream, string method = "POST", CancellationToken cancellationToken = default);
          Task<StreamDownloadResult> DownloadAsync(string route, string method = "GET", CancellationToken cancellationToken = default);
      }
  

  2. StreamService<TService> in <Namespace>.HttpApi.Client

  public class StreamService<TService> : IStreamService<TService>
      {
          private readonly HttpClient _httpClient;
          private readonly IJsonSerializer _jsonSerializer;
          private readonly AbpHttpClientOptions _abpHttpClientOptions;
          private readonly AbpRemoteServiceOptions _abpRemoteServiceOptions;
  
          public StreamService(HttpClient httpClient, IJsonSerializer jsonSerializer, IOptions<AbpHttpClientOptions> abpHttpClientOptions, IOptions<AbpRemoteServiceOptions> abpRemoteServiceOptions)
          {
              _httpClient = httpClient;
              _jsonSerializer = jsonSerializer;
              _abpHttpClientOptions = abpHttpClientOptions.Value;
              _abpRemoteServiceOptions = abpRemoteServiceOptions.Value;
          }
  
          public async Task<StreamDownloadResult> DownloadAsync(string route, string method = "GET", CancellationToken cancellationToken = default)
          {
              StreamDownloadResult result = new StreamDownloadResult();
              var response = await makeRequest(route, null, method, cancellationToken);
              result.ContentType = response.Content.Headers.ContentType.ToString();
              result.ContentLength = response.Content.Headers.ContentLength.Value;
              result.FileName = response.Content.Headers.ContentDisposition.FileName;
              result.Stream = await response.Content.ReadAsStreamAsync();
              return result;
          }
  
          public async Task UploadAsync(string route, Stream stream, string method = "POST", CancellationToken cancellationToken = default)
          {
              await makeRequest(route, new StreamContent(stream), method, cancellationToken);
          }
          private async Task ThrowExceptionForResponseAsync(HttpResponseMessage response)
          {
              if (response.Headers.Contains("_AbpErrorFormat"))
              {
                  IJsonSerializer jsonSerializer = _jsonSerializer;
                  throw new AbpRemoteCallException(jsonSerializer.Deserialize<RemoteServiceErrorResponse>(await response.Content.ReadAsStringAsync(), true).Error);
              }
              var text = await response.Content.ReadAsStringAsync();
              if (text.Contains(":"))
              {
                  text = text.Substring(text.IndexOf(':') + 1);
                  if (text.Contains("\r\n   "))
                  {
                      text = text.Substring(0, text.IndexOf("\r\n   "));
                      throw new AbpRemoteCallException(new RemoteServiceErrorInfo() { Message = text });
                  }
              }
              throw new AbpException($"Remote service returns error! HttpStatusCode: {response.StatusCode}, ReasonPhrase: {response.ReasonPhrase}");
          }
          private async Task<HttpResponseMessage> makeRequest(string route, HttpContent content, string method, CancellationToken cancellationToken)
          {
              var serviceType = typeof(TService);
              DynamicHttpClientProxyConfig clientConfig = _abpHttpClientOptions.HttpClientProxies.GetOrDefault(serviceType) ?? throw new AbpException("Could not get DynamicHttpClientProxyConfig for " + serviceType.FullName + ".");
              RemoteServiceConfiguration remoteServiceConfig = _abpRemoteServiceOptions.RemoteServices.GetConfigurationOrDefault(clientConfig.RemoteServiceName);
              string url = remoteServiceConfig.BaseUrl.EnsureEndsWith('/') + route.TrimStart('/');
              var requestMessage = new HttpRequestMessage(new HttpMethod(method), url) { Content = content };
              HttpResponseMessage response = await _httpClient.SendAsync(requestMessage, HttpCompletionOption.ResponseHeadersRead, cancellationToken);
              if (!response.IsSuccessStatusCode) await ThrowExceptionForResponseAsync(response);
              return response;
          }
      }
  

  3. StreamDownloadResult in <Namespace>.HttpApi.Client

      public class StreamDownloadResult
      {
          public Stream Stream { get; set; }
          public long ContentLength { get; set; }
          public string ContentType { get; set; }
          public string FileName { get; set; }
      }
  

  4. HttpClientAUthenticatorMessageHandler<TService> in <Namespace>.HttpApi.Client

      public class HttpClientAuthenticatorMessageHandler<TService> : DelegatingHandler
      {
          private readonly AbpHttpClientOptions _abpHttpClientOptions;
          private readonly AbpRemoteServiceOptions _abpRemoteServiceOptions;
          private readonly IHttpClientFactory _httpClientFactory;
          private readonly IServiceProvider _serviceProvider;
  
          public HttpClientAuthenticatorMessageHandler(IHttpClientFactory httpClientFactory, IOptions<AbpHttpClientOptions> abpHttpClientOptions, IOptions<AbpRemoteServiceOptions> abpRemoteServiceOptions, IServiceProvider serviceProvider)
          {
              _abpHttpClientOptions = abpHttpClientOptions.Value;
              _abpRemoteServiceOptions = abpRemoteServiceOptions.Value;
              _httpClientFactory = httpClientFactory;
              _serviceProvider = serviceProvider;
          }
  
          protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
          {
              var remoteServiceHttpClientAuthenticator = _serviceProvider.GetService<IRemoteServiceHttpClientAuthenticator>();
              var serviceType = typeof(TService);
              DynamicHttpClientProxyConfig clientConfig = _abpHttpClientOptions.HttpClientProxies.GetOrDefault(serviceType) ?? throw new AbpException("Could not get DynamicHttpClientProxyConfig for " + serviceType.FullName + ".");
              RemoteServiceConfiguration remoteServiceConfig = _abpRemoteServiceOptions.RemoteServices.GetConfigurationOrDefault(clientConfig.RemoteServiceName);
              using (var client = _httpClientFactory.CreateClient())
              {
                  await remoteServiceHttpClientAuthenticator.Authenticate(new RemoteServiceHttpClientAuthenticateContext(client, request, remoteServiceConfig, clientConfig.RemoteServiceName));
                  foreach (var header in client.DefaultRequestHeaders)
                      request.Headers.Add(header.Key, header.Value);
                  return await base.SendAsync(request, cancellationToken);
              }
          }
      }
  

  5. Configure your stream service in <Namespace>.HttpApi.Client module

          public static void ConfigureStreamServices(ServiceConfigurationContext context)
          {
              context.Services.AddTransient<HttpClientAuthenticatorMessageHandler<YOUR APPLICATION SERVICE>>();
              context.Services.AddHttpClient<IStreamService<YOUR APPLICATION SERVICE>, StreamService<YOUR APPLICATION SERVICE>>()
                  .AddHttpMessageHandler<HttpClientAuthenticatorMessageHandler<YOUR APPLICATION SERVICE>>();
          }
  

  6. Make use of IStreamService<YOUR APPLICATION SERVICE> using Dependency Injection in one of your classes:

  public ClassConstructor(..., IStreamService<YOUR APPLICATION SERVICE> streamService, ...) 
  { 
  ...
  _streamService = streamService;
  ...
  }
  

          public async Task RunAsync()
          {
              var id = Guid.NewGuid();
              string fileName = "asd";
              await _streamService.UploadAsync($"/api/app/APP SERVICE PATH/upload/{id}", new FileStream(fileName, FileMode.Open));
              var text = File.ReadAllText(fileName);
              var dlStream = await _streamService.DownloadAsync($"/api/app/APP SERVICE PATH/download/{id}");
              var reader = new StreamReader(dlStream.Stream);
              var dlText = reader.ReadToEnd();
              if (text != dlText)
              {
                  throw new Exception("Uploaded log does not match downloaded log.");
              }
          }
  

  Save Cancel

  1
• 

  0

  alexandru-bagu created 3 years ago

  Do I get my question back now that I answered my own question?

  Save Cancel
• 

  0

  alper created 3 years ago

  Support Team Director

  your credit has been refunded.

  Save Cancel
• 

  0

  alper created 3 years ago

  Support Team Director

  Stream support for the application service methods has been completed. You can test the preview version of 3.3.0

  https://github.com/abpframework/abp/blob/dev/docs/en/Blog-Posts/2020-10-15%20v3_3_Preview/POST.md#stream-support-for-the-application-service-methods

  The production version will be released 2 weeks later.

  Save Cancel
• 

  0

  hikalkan created 3 years ago

  Support Team Co-Founder

  Thanks a lot @alexandru-bagu for your great contribution. It is already documented: https://docs.abp.io/en/abp/3.3/Application-Services#working-with-streams Closing this question.

  Save Cancel
• 

  0

  jtallon created 3 years ago

  @hikalkan does this work with the FileManager Pro-Module?

  Can't upload large files and in the modern world, this makes the File Manager module almost worthless as a pro-module?

  Save Cancel