Open Closed

Data Seeding in monolith and microservice. #6950


User avatar
0
cangunaydin created
  • ABP Framework version: v8.0.0
  • Database System: EF Core (PostgreSQL)
  • Tiered (for MVC) or Auth Server Separated (for Angular): yes

Hello, I have a monolith abp application, which i created separate modules inside. Now i have created host application on one of my modules to use it as a microservice.

While i am converting the app to support microservice. I have realized that monolith app use IDataSeeder for data seeding then all the classes that implements IDataSeedContributor is contributing to data seed process. And this is working perfectly. For ex: when i create a new tenant, it also creates the admin user, and give permissions.

But when i was looking at the abp microservice startup template i have seen that, for identity,openiddict,permissions and language there are different data seeders. These implementations are custom implementations and do not use IDataSeeder. so here is my questions.

  • Is it possible to use IDataSeeder in microservice projects to seed data? For ex, for identity data seeding, if your microservice includes identity module you can just call _dataSeeder.Seed() method with a context and all the contributors should contribute to data seeding. Why this option is not used? As i see from microservice template, all the data seeding has been done one by one.

  • to test the idea i have injected IDataSeeder to event handler class. When tenant is created, the SeedAsync() method triggered. User has been created but there has been some problems with permissions (permissions are not created for the admin user of the new tenant). So another question regarding the situation, how does monolith app seeding the permissions for new tenant's admin with _dataSeeder.SeedAsync() method? And why it doesn't work on microservice startup template?

Thank you for the assistance.


12 Answer(s)
  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    Have you checked this documentation?

    https://docs.abp.io/en/commercial/latest/startup-templates/microservice/database-migrations

  • User Avatar
    0
    cangunaydin created

    Hello again, yes i have read that, but the question over here is

    Why IDataSeeder is not directly triggered even on the fly migrations? Cause the microservice already includes those classes (classes that implements IDataSeedContributor)?

    To examplify it. Why don't we directly trigger IDataSeeder like i have shown here, when the tenant is created.

    Or another way of asking this is, why monolith app can seed the identity data with IDataSeeder and Identity MicroService can not with the same method? (so we need to use IIdentityDataSeeder explicitly)

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    I will forward this to our colleague.

  • User Avatar
    0
    cangunaydin created

    ok thank you @maliming

  • User Avatar
    0
    gterdem created
    Support Team Senior .NET Developer

    Hello cangunaydin,

    Basically, the IDataSeeder (https://github.com/abpframework/abp/blob/dev/framework/src/Volo.Abp.Data/Volo/Abp/Data/DataSeeder.cs) finds all the related IDataSeedContributors in the DI container and runs the SeedAsync method.

    Some of the modules need some initial data in order to operate. Hence, they have their own DataSeedContributor, like IdentityDataSeedContributor or PermissionDataSeedContributor. Or you may have your own business related DataSeedContributor. All of these DataSeedContributors are added to DI and seeded via IDataSeeder. Notice that all of them are located under the Domain layer of the layered architectured module (or the application).

    In the monolith applications, IDataSeeder seeds the modules' DataSeedContributors because they are directly or indirectly referenced to the Domain of that module. That means they are already registered to the Dependency Injection Container.

    However in a microservice architecture, the modules are seperated into different microservices. All the microservices doesn't have references to all the modules. That's why you need to create different DataSeeders (like the IdentityDataSeeder in your example) to handle the seeding manually.

  • User Avatar
    0
    cangunaydin created

    Hello @gterdem, That's exactly what i think, and that's why i raised the question,

    If we continue from microservice example, for ex, in identityservice (Microservice) you have reference to identiy module, you indirectly referencing idataseedcontributor of the identity module. So why not call _dataseeder.Seed() method directly instead explicit calls to different dataseed service? I think you want to seed the data of the related microservice isn't it (which includes related modules)?

    For ex, IdentityService (in Microservice Template) includes (Volo.Abp.Identity.Pro.Domain and Volo.Abp.OpenIddict.Pro.Domain) which also includes related DataSeedContributors (for ex, IdentitySeedContributor) coming from those modules. Am i missing sth over here?

    here is the second part of my question

    Let's assume that i create one microservice containing all abp modules,(which was my monolith app). Let's name this MainService. Then created extra microservice. Let's name it ExtraService. In this case if i call IDataSeeder.Seed() method when new tenant is created according to your explanation everything should work fine. And MainService should create admin user and permissions. But i am seeing that permissions are not created. What can be the reason for that?

  • User Avatar
    0
    gterdem created
    Support Team Senior .NET Developer

    First of all IdentityDataSeedContributor is basically the same with the application template. The idea is as you can see, to override the default AdminEmail and AdminPassword.

    Second, Permissions are located under the Application.Contracts layer and the PermissionDefinitionProvider registers the permissions. Which means you need to find the permissions first to seed them.

    In the microservice template, this is a sample of how the permissions are seeded in the AdministrationServiceDataSeeder:

    public async Task SeedAsync(Guid? tenantId = null)
        {
            using (_currentTenant.Change(tenantId))
            {
                using (var uow = _unitOfWorkManager.Begin(requiresNew: true, isTransactional: true))
                {
                    var multiTenancySide = tenantId == null
                        ? MultiTenancySides.Host
                        : MultiTenancySides.Tenant;
    
                    var permissionNames = (await _permissionDefinitionManager
                        .GetPermissionsAsync())
                        .Where(p => p.MultiTenancySide.HasFlag(multiTenancySide))
                        .Where(p => !p.Providers.Any() || p.Providers.Contains(RolePermissionValueProvider.ProviderName))
                        .Select(p => p.Name)
                        .ToArray();
                    
                    _logger.LogInformation($"Seeding admin permissions.");
                    await _permissionDataSeeder.SeedAsync(
                        RolePermissionValueProvider.ProviderName,
                        "admin",
                        permissionNames,
                        tenantId
                    );
                    
                    _logger.LogInformation($"Seeding language data.");
                    await _languageManagementDataSeeder.SeedAsync();
    
                    await uow.CompleteAsync();
                }
            }
        }
    

    Why? Because it is using dynamic permission store which means each microservice is writing their own permissions to the AdministrationService database directly and the dataseeder is granting these permissions. Otherwise, AdministrationService will need to reference to all the microservices' Application.Contracts layers (which it was, before we implemented the dynamic permission store).

    See PermissionDataSeedContributor and PermissionDataSeeder for more details.

  • User Avatar
    0
    cangunaydin created

    Hello @gterdem, Thank you for the answer, but i think we are talking different things. I will try to explain what i am thinking step by step, hope i can manage that. I will talk about AdministrationService inside microservice template and how do I think about it.

    • PermissionManagement Module is part of administration service
    • So it includes PermissionDataSeedContributor that you have sent the link in the previous message.
    • When application started it registers the PermissionDataSeedContributor to the dependency injection system.
    • When you compare AdministrationServiceDataSeeder (omitting language management for now) and PermissionDataSeedContributor they are exactly doing same thing (correct me if i am wrong over here or i miss sth)

    PermissionDataSeedContributor

    
      var multiTenancySide = CurrentTenant.GetMultiTenancySide();
            var permissionNames = (await PermissionDefinitionManager.GetPermissionsAsync())
                .Where(p => p.MultiTenancySide.HasFlag(multiTenancySide))
                .Where(p => !p.Providers.Any() || p.Providers.Contains(RolePermissionValueProvider.ProviderName))
                .Select(p => p.Name)
                .ToArray();
    
            await PermissionDataSeeder.SeedAsync(
                RolePermissionValueProvider.ProviderName,
                "admin",
                permissionNames,
                context?.TenantId
            );
    
    

    AdministrationServiceDataSeeder

    var multiTenancySide = tenantId == null
        ? MultiTenancySides.Host
        : MultiTenancySides.Tenant;
    
    var permissionNames = (await _permissionDefinitionManager
        .GetPermissionsAsync())
        .Where(p => p.MultiTenancySide.HasFlag(multiTenancySide))
        .Where(p => !p.Providers.Any() || p.Providers.Contains(RolePermissionValueProvider.ProviderName))
        .Select(p => p.Name)
        .ToArray();
        
     _logger.LogInformation($"Seeding admin permissions.");
     await _permissionDataSeeder.SeedAsync(
         RolePermissionValueProvider.ProviderName,
         "admin",
         permissionNames,
         tenantId
     );
    
    • So as a conclusion, if i inject IDataSeeder to AdministrationServiceDataSeeder and call _dataSeeder.SeedAsync(), i am expecting the _dataSeeder.SeedAsync() should trigger PermissionDataSeedContributor, and seeding the data.

    what do i miss over here? can you point out? look at the image below, why this code won't work?


    Since i am curious about the question above, I do not use microservice template, i have a different setup. In my setup all abp modules are in one microservice, So i don't have any separated abp modules in different microservices, I am expecting data seeding to seed default static permissions but IDataSeeder not seeding the permission data when i create new tenant. That's why i raised the question at the first place. But if you can answer the above i suppose it would help with my problem.

  • User Avatar
    0
    gterdem created
    Support Team Senior .NET Developer

    I think I see your point. The reason can be the PermissionDefinitionManager not able to find the permissions in the database yet because it would be running very first in the module initializations.

    Can you make sure that the tenant permissions exist in the database when you run the dataseeder as you shared in the screenshot?

  • User Avatar
    0
    cangunaydin created

    Hello again, i don't think it is related with PermissionDefinitionManager. I did some tests. Actually Permissions are seeded in the db. But i am getting error on the ui side when i try to login here is the video for that. What i did is just to change dataseeding as i mentioned before.

    https://drive.google.com/file/d/1cXbxgi6WZF1_m7c9d8iDbqd49lpCU1C2/view?usp=sharing

    here is the code that i have changed, when the new tenant has been created it triggers.

    using System;
    using System.Collections.Generic;
    using System.Threading.Tasks;
    using ApproveIt.Shared.Hosting.Microservices.DbMigrations;
    using Doohlink.EntityFrameworkCore;
    using Microsoft.Extensions.Logging;
    using Volo.Abp.Data;
    using Volo.Abp.EventBus.Distributed;
    using Volo.Abp.Identity;
    using Volo.Abp.MultiTenancy;
    using Volo.Abp.Uow;
    using Volo.Saas.Tenants;
    
    namespace Doohlink.DbMigrations;
    
    public class AdzupDatabaseMigrationEventHandler
        : DatabaseMigrationEventHandlerBase<DoohlinkDbContext>,
            IDistributedEventHandler<TenantCreatedEto>,
            IDistributedEventHandler<TenantConnectionStringUpdatedEto>,
            IDistributedEventHandler<ApplyDatabaseMigrationsEto>
    {
        private readonly IdentityServiceDataSeeder _identityServiceDataSeeder;
        private readonly AdministrationServiceDataSeeder _administrationServiceDataSeeder;
        private readonly IDataSeeder _dataSeeder;
    
        public AdzupDatabaseMigrationEventHandler(
            ILoggerFactory loggerFactory,
            ICurrentTenant currentTenant,
            IUnitOfWorkManager unitOfWorkManager,
            ITenantStore tenantStore,
            ITenantRepository tenantRepository,
            IDistributedEventBus distributedEventBus,
            IdentityServiceDataSeeder identityServiceDataSeeder,
            AdministrationServiceDataSeeder administrationServiceDataSeeder,
            IDataSeeder dataSeeder) : base(
            loggerFactory,
            currentTenant,
            unitOfWorkManager,
            tenantStore,
            tenantRepository,
            distributedEventBus,
            DoohlinkDbProperties.ConnectionStringName)
        {
            _identityServiceDataSeeder = identityServiceDataSeeder;
            _administrationServiceDataSeeder = administrationServiceDataSeeder;
            _dataSeeder = dataSeeder;
        }
    
        public async Task HandleEventAsync(ApplyDatabaseMigrationsEto eventData)
        {
            if (eventData.DatabaseName != DatabaseName)
            {
                return;
            }
    
            try
            {
                var schemaMigrated = await MigrateDatabaseSchemaAsync(eventData.TenantId);
                await _identityServiceDataSeeder.SeedAsync(
                    tenantId: eventData.TenantId,
                    adminEmail: DoohlinkConsts.AdminEmailDefaultValue,
                    adminPassword: DoohlinkConsts.AdminPasswordDefaultValue
                );
                await _administrationServiceDataSeeder.SeedAsync(eventData.TenantId);
    
                if (eventData.TenantId == null && schemaMigrated)
                {
                    /* Migrate tenant databases after host migration */
                    await QueueTenantMigrationsAsync();
                }
            }
            catch (Exception ex)
            {
                await HandleErrorOnApplyDatabaseMigrationAsync(eventData, ex);
            }
        }
    
        public async Task HandleEventAsync(TenantCreatedEto eventData)
        {
            try
            {
                await MigrateDatabaseSchemaAsync(eventData.Id);
    
                await _dataSeeder.SeedAsync(
                    new DataSeedContext(eventData.Id)
                        .WithProperty(IdentityDataSeedContributor.AdminEmailPropertyName,
                            DoohlinkConsts.AdminEmailDefaultValue)
                        .WithProperty(IdentityDataSeedContributor.AdminPasswordPropertyName,
                            DoohlinkConsts.AdminPasswordDefaultValue)
                );
                // await _identityServiceDataSeeder.SeedAsync(
                //     tenantId: eventData.Id,
                //     adminEmail: eventData.Properties.GetOrDefault(IdentityDataSeedContributor.AdminEmailPropertyName) ??
                //                 DoohlinkConsts.AdminEmailDefaultValue,
                //     adminPassword:
                //     eventData.Properties.GetOrDefault(IdentityDataSeedContributor.AdminPasswordPropertyName) ??
                //     DoohlinkConsts.AdminPasswordDefaultValue
                // );
                // await _administrationServiceDataSeeder.SeedAsync(eventData.Id);
            }
            catch (Exception ex)
            {
                await HandleErrorTenantCreatedAsync(eventData, ex);
            }
        }
    
        public async Task HandleEventAsync(TenantConnectionStringUpdatedEto eventData)
        {
            if (eventData.ConnectionStringName != DatabaseName &&
                eventData.ConnectionStringName != ConnectionStrings.DefaultConnectionStringName ||
                eventData.NewValue.IsNullOrWhiteSpace())
            {
                return;
            }
    
            try
            {
                await MigrateDatabaseSchemaAsync(eventData.Id);
                await _identityServiceDataSeeder.SeedAsync(
                    tenantId: eventData.Id,
                    adminEmail: DoohlinkConsts.AdminEmailDefaultValue,
                    adminPassword: DoohlinkConsts.AdminPasswordDefaultValue
                );
                await _administrationServiceDataSeeder.SeedAsync(eventData.Id);
                /* You may want to move your data from the old database to the new database!
                 * It is up to you. If you don't make it, new database will be empty
                 * (and tenant's admin password is reset to IdentityServiceDbProperties.DefaultAdminPassword). */
            }
            catch (Exception ex)
            {
                await HandleErrorTenantConnectionStringUpdatedAsync(eventData, ex);
            }
        }
    }
    
    

    if i uncomment the lines that is commented(if i revert it like in microservice template), it works fine. What can be the reason i wonder? and i can see the permissions that is seeded in db for newly created tenant. (for idataseeder)

  • User Avatar
    0
    cangunaydin created

    I am also having some problems with localizations. I have 2 hosts that is running on my cluster. Let's calll them "Main Host" and "Side Host". Localizations are stored on Main Db. If I start the servers first time all the localizations are working fine. Then after some time if i restart the side server, all the "side" localizations are gone from UI. To fix that i need to flush the redis. Then everything is fixed again. What can be the reason for it? I removed DynamicLocalizationResourceContributor from both hosts. So i was suspicious about that, but even if revert it and try it again. It doesn't work. On application start if redis cache is empty, it gets it from db and hold it in the server memory i guess. The problem is related with redis i believe? and if i remove DynamicLocalizationResourceContributor does it mean it longer use Redis? or it doesn't write to redis or it doesn't read or write to redis?

  • User Avatar
    0
    ahmetfarukulu created

    if i uncomment the lines that is commented(if i revert it like in microservice template), it works fine. What can be the reason i wonder? and i can see the permissions that is seeded in db for newly created tenant. (for idataseeder)

    You can use the IDataSeeder just like monolith application templates if you want. However, we explicitly seed data for each microservice because a race condition could occur during the seeding process. For example, every microservice has a dependency on the Permission Entity Framework package. When it tries to retrieve a list of existing permissions in PermissionDataSeeder, two or more microservices execute the same point and insert the same permissions. IDataSeeder collects all contributors, and we can't limit the contributor types easily. That's why creating a custom seeding for each microservice is a more suitable solution.

    I created an empty microservice project and changed the data seeder just like you did. When I create the new tenant, there isn't any problem; everything works as expected. Can you share the logs when you get the exception?

    I removed DynamicLocalizationResourceContributor from both hosts. So i was suspicious about that, but even if revert it and try it again. It doesn't work. On application start if redis cache is empty, it gets it from db and hold it in the server memory i guess. The problem is related with redis i believe? and if i remove DynamicLocalizationResourceContributor does it mean it longer use Redis? or it doesn't write to redis or it doesn't read or write to redis?

    When the application starts, it tries to save existing localization files to the AbpLocalizationTexts table and caches these files in Redis. So, if you run both applications and clear the cache afterward, you can run one of your applications, and there shouldn't be any problem; all localizations are already inserted into the database, and it caches all localizations as expected. Make sure both application has Language Managemet Entity Framework dependency.

Made with ❤️ on ABP v8.2.0-preview Updated on March 25, 2024, 15:11