ConcurrentModificationException in ReportingServiceManager#downloadFileAsync

[markusheiden]

Recently we got a ConcurrentModificationException in the ReportingServiceManager.

IMO that cannot be caused by our code, because we use a new ReportingServiceManager each time.

Maybe there is a problem in JAXB or CXF. Currently, we are using org.glassfish.jaxb 4.0.3 and org.apache.cxf 4.0.2.

Or the ReportingServiceManager has a concurrency issue.

Our code:

new ReportingServiceManager(authorizationData, environment)
    .downloadFileAsync(parameters, null)
    .get(10, MINUTES);

Stacktrace:

java.util.ConcurrentModificationException: null
    at java.util.HashMap.computeIfAbsent
    at org.glassfish.jaxb.runtime.v2.model.impl.TypeInfoSetImpl.getXmlNs(TypeInfoSetImpl.java:298)
    at org.glassfish.jaxb.runtime.v2.schemagen.XmlSchemaGenerator$Namespace.writeTo(XmlSchemaGenerator.java:565)
    at org.glassfish.jaxb.runtime.v2.schemagen.XmlSchemaGenerator.write(XmlSchemaGenerator.java:415)
    at org.glassfish.jaxb.runtime.v2.runtime.JAXBContextImpl.generateSchema(JAXBContextImpl.java:770)
    at org.apache.cxf.common.jaxb.JAXBUtils.generateJaxbSchemas(JAXBUtils.java:810)
    at org.apache.cxf.jaxb.JAXBDataBinding.generateJaxbSchemas(JAXBDataBinding.java:474)
    at org.apache.cxf.jaxb.JAXBDataBinding.initialize(JAXBDataBinding.java:391)
    at org.apache.cxf.service.factory.AbstractServiceFactoryBean.initializeDataBindings(AbstractServiceFactoryBean.java:87)
    at org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.buildServiceFromClass(ReflectionServiceFactoryBean.java:469)
    at org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.buildServiceFromClass(JaxWsServiceFactoryBean.java:693)
    at org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.initializeServiceModel(ReflectionServiceFactoryBean.java:529)
    at org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.create(ReflectionServiceFactoryBean.java:262)
    at org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.create(JaxWsServiceFactoryBean.java:199)
    at org.apache.cxf.jaxws.ServiceImpl.createPort(ServiceImpl.java:466)
    at org.apache.cxf.jaxws.ServiceImpl.getPort(ServiceImpl.java:342)
    at org.apache.cxf.jaxws.ServiceImpl.getPort(ServiceImpl.java:337)
    at jakarta.xml.ws.Service.getPort(Service.java:210)
    at com.microsoft.bingads.internal.ServiceFactoryImpl.createProxyFromService(ServiceFactoryImpl.java:179)
    at com.microsoft.bingads.ServiceClient.getService(ServiceClient.java:159)
    at com.microsoft.bingads.v13.reporting.ReportingServiceManager.submitDownloadAsync(ReportingServiceManager.java:223)
    at com.microsoft.bingads.v13.reporting.ReportingServiceManager.downloadFileAsyncImpl(ReportingServiceManager.java:153)
    at com.microsoft.bingads.v13.reporting.ReportingServiceManager.downloadFileAsync(ReportingServiceManager.java:145)
    ...

[qitia]

hi, thanks for reaching out.

May I know what is the repro rate of the issue from your side?

Seems to me create a new ReportServiceManager instance for each call is not a good practice. Did you try reuse a single ReportServiceManager?

[markusheiden]

I just got this error once.

I will try to reuse the ReportingServiceManager as much as possible. We are using different accounts, so reuse is not always possible.

[markusheiden]

Another one, this time in the BulkServiceManager:

java.util.ConcurrentModificationException: null
    at java.util.HashMap.computeIfAbsent
    at org.glassfish.jaxb.runtime.v2.model.impl.TypeInfoSetImpl.getXmlNs(TypeInfoSetImpl.java:298)
    at org.glassfish.jaxb.runtime.v2.schemagen.XmlSchemaGenerator$Namespace.writeTo(XmlSchemaGenerator.java:565)
    at org.glassfish.jaxb.runtime.v2.schemagen.XmlSchemaGenerator.write(XmlSchemaGenerator.java:415)
    at org.glassfish.jaxb.runtime.v2.runtime.JAXBContextImpl.generateSchema(JAXBContextImpl.java:770)
    at org.apache.cxf.common.jaxb.JAXBUtils.generateJaxbSchemas(JAXBUtils.java:810)
    at org.apache.cxf.jaxb.JAXBDataBinding.generateJaxbSchemas(JAXBDataBinding.java:474)
    at org.apache.cxf.jaxb.JAXBDataBinding.initialize(JAXBDataBinding.java:391)
    at org.apache.cxf.service.factory.AbstractServiceFactoryBean.initializeDataBindings(AbstractServiceFactoryBean.java:87)
    at org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.buildServiceFromClass(ReflectionServiceFactoryBean.java:469)
    at org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.buildServiceFromClass(JaxWsServiceFactoryBean.java:693)
    at org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.initializeServiceModel(ReflectionServiceFactoryBean.java:529)
    at org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.create(ReflectionServiceFactoryBean.java:262)
    at org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.create(JaxWsServiceFactoryBean.java:199)
    at org.apache.cxf.jaxws.ServiceImpl.createPort(ServiceImpl.java:466)
    at org.apache.cxf.jaxws.ServiceImpl.getPort(ServiceImpl.java:342)
    at org.apache.cxf.jaxws.ServiceImpl.getPort(ServiceImpl.java:337)
    at jakarta.xml.ws.Service.getPort(Service.java:210)
    at com.microsoft.bingads.internal.ServiceFactoryImpl.createProxyFromService(ServiceFactoryImpl.java:179)
    at com.microsoft.bingads.ServiceClient.getService(ServiceClient.java:159)
    at com.microsoft.bingads.v13.bulk.BulkServiceManager.submitDownloadAsync(BulkServiceManager.java:559)
    at com.microsoft.bingads.v13.bulk.BulkServiceManager.downloadFileAsyncImpl(BulkServiceManager.java:484)
    at com.microsoft.bingads.v13.bulk.BulkServiceManager.downloadEntitiesAsyncImpl(BulkServiceManager.java:435)
    at com.microsoft.bingads.v13.bulk.BulkServiceManager.downloadEntitiesAsync(BulkServiceManager.java:163)

[markusheiden]

For me this looks like a concurrency issue in the service managers:
jakarta.xml.ws.Service and its implementation are not marked explicitly as thread-safe. So the com.microsoft.bingads.ServiceClient that wraps this class, may not be used from different threads. This seems to not play well with the asynchronous methods in the service managers.

The only suspicious location I found, was the com.microsoft.bingads.internal.ServiceFactoryImpl that creates a new jakarta.xml.ws.Service using a thread pool. Though it shuts down the thread pool immediately after the creation it does not wait for the shutdown to complete.

I provided #174 to fix this issue.

[IRus]

We just got the same exception on 13.0.24.3 (cxf 4.0.8):

java.util.ConcurrentModificationException: null
    at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1013)
    at java.util.ArrayList$Itr.next(ArrayList.java:967)
    at org.apache.cxf.jaxrs.provider.ProviderFactory.createMessageBodyWriter(ProviderFactory.java:570)
    at org.apache.cxf.jaxrs.provider.ProviderFactory.createMessageBodyWriterInterceptor(ProviderFactory.java:465)
    at org.apache.cxf.jaxrs.client.AbstractClient.writeBody(AbstractClient.java:519)
    at org.apache.cxf.jaxrs.client.WebClient$BodyWriter.doWriteBody(WebClient.java:1222)
    at org.apache.cxf.jaxrs.client.AbstractClient$AbstractBodyWriter.handleMessage(AbstractClient.java:1223)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
    at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:710)
    at org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1085)
    at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:931)
    at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:900)
    at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:460)
    at org.apache.cxf.jaxrs.client.SyncInvokerImpl.method(SyncInvokerImpl.java:150)
    at org.apache.cxf.jaxrs.client.SyncInvokerImpl.method(SyncInvokerImpl.java:145)
    at org.apache.cxf.jaxrs.client.SyncInvokerImpl.put(SyncInvokerImpl.java:100)
    at org.apache.cxf.jaxrs.client.spec.InvocationBuilderImpl.put(InvocationBuilderImpl.java:167)
    at com.microsoft.bingads.internal.restful.RestfulServiceClient.getResponseInfo(RestfulServiceClient.java:200)
    at com.microsoft.bingads.internal.restful.CampaignManagementService.sendRequest(CampaignManagementService.java:89)
    at com.microsoft.bingads.internal.restful.CampaignManagementService.updateCampaigns(CampaignManagementService.java:221)
    ...
    at reactor.core.publisher.MonoRunnable.subscribe(MonoRunnable.java:49)
    at reactor.core.publisher.Mono.subscribe(Mono.java:4495)
    at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:427)
    at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.runAsync(FluxPublishOn.java:440)
    at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.run(FluxPublishOn.java:527)
    at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
    at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
    at java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.lang.Thread.run(Thread.java:840)

jakarta.ws.rs.ProcessingException: java.util.ConcurrentModificationException
    at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:638)
    at org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:612)
    at org.apache.cxf.jaxrs.client.WebClient.doResponse(WebClient.java:1149)
    at org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1086)
    at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:931)
    at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:900)
    at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:460)
    at org.apache.cxf.jaxrs.client.SyncInvokerImpl.method(SyncInvokerImpl.java:150)
    at org.apache.cxf.jaxrs.client.SyncInvokerImpl.method(SyncInvokerImpl.java:145)
    at org.apache.cxf.jaxrs.client.SyncInvokerImpl.put(SyncInvokerImpl.java:100)
    at org.apache.cxf.jaxrs.client.spec.InvocationBuilderImpl.put(InvocationBuilderImpl.java:167)
    at com.microsoft.bingads.internal.restful.RestfulServiceClient.getResponseInfo(RestfulServiceClient.java:200)
    at com.microsoft.bingads.internal.restful.CampaignManagementService.sendRequest(CampaignManagementService.java:89)
    at com.microsoft.bingads.internal.restful.CampaignManagementService.updateCampaigns(CampaignManagementService.java:221)
    ...
    at reactor.core.publisher.MonoRunnable.subscribe(MonoRunnable.java:49)
    at reactor.core.publisher.Mono.subscribe(Mono.java:4495)
    at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:427)
    at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.runAsync(FluxPublishOn.java:440)
    at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.run(FluxPublishOn.java:527)
    at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
    at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
    at java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.lang.Thread.run(Thread.java:840)

Related to #203 as well

[IRus]

Related to #184

[IRus]

Dennis Yemelyanov (@dennis-yemelyanov) do you something can be done on you side? Or issue for CXF should be created?